1
|
Turner DA, Poitevien P. Foreword: Preparing future pediatricians and pediatric subspecialists: Competency-based medical education - Part 2. Curr Probl Pediatr Adolesc Health Care 2024; 54:101676. [PMID: 39142927 DOI: 10.1016/j.cppeds.2024.101676] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 08/16/2024]
Affiliation(s)
- David A Turner
- Competency-Based Medical Education, American Board of Pediatrics, Chapel Hill, NC, USA.
| | - Patricia Poitevien
- Equity, and Inclusion, The Warren Alpert Medical School of Brown University, Providence, RI, USA
| |
Collapse
|
2
|
Bowen J, Polak C, Thomson J, Herrmann L. A Pilot Longitudinal Clinical Reasoning Curriculum for Pediatric Residents. MEDEDPORTAL : THE JOURNAL OF TEACHING AND LEARNING RESOURCES 2024; 20:11447. [PMID: 39323975 PMCID: PMC11422513 DOI: 10.15766/mep_2374-8265.11447] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/08/2024] [Accepted: 05/09/2024] [Indexed: 09/27/2024]
Abstract
Introduction Clinical reasoning (CR) is required for physicians. Pediatric residents often gain CR skills through experiential learning. Currently, deliberate education on CR targeted toward pediatric residents is inconsistent. Our objective was to implement a pilot CR curriculum, including five hour-long sessions, and evaluate its impact on self-identified CR Milestones and comfort with CR skills. Methods We used Kern's six steps for curriculum development to develop our curriculum. Five morning report sessions included didactics and small-group activities. Pre/post surveys assessed resident self-identified level on ACGME Milestones related to CR skills (Patient Care 4 [PC4] and Medical Knowledge 2 [MK2]) and comfort with CR skills. The postsurvey assessed resident attitudes toward the sessions. Paired samples for Milestone and comfort-based questions were analyzed using Wilcoxon signed rank tests. Attitude questions were reported with descriptive statistics. Results Each of the five curricular sessions was attended by 40-50 pediatric residents. Seventy-one trainees (58% of residency) and 51 trainees (42% of residency) completed the pre- and postsurveys, respectively, with 20 paired samples. Self-assessment of PC4 (p = .006) and resident comfort with all measured CR skills increased significantly. Of trainees who attended at least one session (n = 44), most reported finding the sessions helpful (97%), relevant to their clinical work (97%), and impactful on their clinical practice (73%). Discussion Following exposure to this CR curriculum, pediatric residents reported increased self-identified competency levels on the evaluated Milestones and improved comfort with CR skills. Dedicated CR education may advance pediatric resident understanding of and comfort with CR.
Collapse
Affiliation(s)
- James Bowen
- Clinical Fellow, Department of Pediatric Hospital Medicine, Cincinnati Children's Hospital Medical Center
| | - Catherine Polak
- Assistant Professor of Pediatrics, Department of Pediatric Hospital Medicine, UPMC Children's Hospital of Pittsburgh
| | - Joanna Thomson
- Associate Professor of Pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine; Attending Physician, Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center
| | - Lisa Herrmann
- Associate Professor of Pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine; Attending Physician, Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center
| |
Collapse
|
3
|
Turner DA, Poitevien P. Foreword: Preparing future pediatricians and pediatric subspecialists: Competency-Based Medical Education-Part 1. Curr Probl Pediatr Adolesc Health Care 2024; 54:101639. [PMID: 38897839 DOI: 10.1016/j.cppeds.2024.101639] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 06/21/2024]
Affiliation(s)
- David A Turner
- Vice President, Competency-Based Medical Education, American Board of Pediatrics, Chapel Hill, NC, United States of America.
| | - Patricia Poitevien
- Dean of Diversity Equity and Inclusion, Brown University, Providence, RI, United States of America
| |
Collapse
|
4
|
Burke AE, Sklansky DJ, Haftel HM, Mitchell A, Mann KJ. Competency-based medical education and the education continuum: Establishing a framework for lifelong learning. Curr Probl Pediatr Adolesc Health Care 2024; 54:101642. [PMID: 38851971 DOI: 10.1016/j.cppeds.2024.101642] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 06/10/2024]
Affiliation(s)
- Ann E Burke
- Wright State University Boonshoft School of Medicine and Dayton Children's Hospital.
| | - Daniel J Sklansky
- University of Wisconsin School of Medicine and Public Health and American Family Children's Hospital
| | - Hilary M Haftel
- American Academy of Pediatrics, Senior Vice President, Education
| | - Andrew Mitchell
- Wright State University Boonshoft School of Medicine and Dayton Children's Hospital
| | - Keith J Mann
- American Board of Pediatrics, Vice President for Continuing Certification
| |
Collapse
|
5
|
Mink RB, Schwartz A, Mahan JD, Fussell JJ, George R, Schumacher DJ, McFadden V, Turner DA, Atlas MP. Level of Supervision for the Entrustable Professional Activities Common to General Pediatrics and the Subspecialties Decreases from Residency to Fellowship. Acad Pediatr 2024; 24:1025-1030. [PMID: 38631477 DOI: 10.1016/j.acap.2024.04.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Revised: 04/06/2024] [Accepted: 04/10/2024] [Indexed: 04/19/2024]
Abstract
OBJECTIVES To compare level of supervision (LOS) ratings of graduating pediatric residents with their assessments as fellows for the five Entrustable Professional Activities (EPAs) common to general pediatrics and the subspecialties and to determine if the difference between ratings from residency to fellowship is less for the QI and Practice Management EPAs, since the skills needed to perform these may be less context-dependent. METHODS We compared ratings of graduating residents with their assessments as fellows using LOS data from two sequential EPA studies. RESULTS There were 65 ratings from 41 residents at the first fellow assessment. At graduation, most residents needed little to no supervision for all EPAs with 94% (61/65) of ratings level four or five. In contrast, only 5/65 (8%) of the first fellow assessments were level four or five. The ratings difference for the QI and Practice Management EPAs was similar to the others. CONCLUSIONS LOS ratings for the EPAs common to generalists and subspecialists reset as residents become fellows. There was no evidence that the QI and Practice Management EPAs are less context-dependent. This study provides additional validity evidence for using these LOS scales to assess trainees in pediatric residency and fellowship.
Collapse
Affiliation(s)
- Richard B Mink
- David Geffen School of Medicine at UCLA (RB Mink), Los Angeles, Calif and the Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, Torrance, Calif.
| | - Alan Schwartz
- The Michael Reese Endowed Professor of Medical Education (A Schwartz), University of Illinois College of Medicine at Chicago, Chicago, Ill
| | - John D Mahan
- Department of Pediatrics (JD Mahan), Pediatric Nephrology Fellowship Program Director, Nationwide Children's Hospital, The Ohio State University College of Medicine, Columbus, Ohio
| | - Jill J Fussell
- University of Arkansas for Medical Sciences and Arkansas Children's Hospital (JJ Fussell), Little Rock, Ark
| | - Roshan George
- Department of Pediatrics (R George), Pediatric Nephrology Fellowship Program Director, Emory University and Children's Healthcare of Atlanta, Atlanta, Ga
| | - Daniel J Schumacher
- Department of Pediatrics (DJ Schumacher), Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Vanessa McFadden
- Department of Pediatrics (V McFadden), Medical College of Wisconsin, Milwaukee, Wis
| | - David A Turner
- Competency-Based Medical Education (DA Turner), American Board of Pediatrics, Chapel Hill, NC
| | - Mark P Atlas
- Zucker School of Medicine at Hofstra-Northwell (MP Atlas), Hempstead, NY and Head, Neuro-Oncology, Cohen Children's Medical Center, New Hyde Park, NY
| |
Collapse
|
6
|
Kulasegaram KM, Grierson L, Barber C, Chahine S, Chou FC, Cleland J, Ellis R, Holmboe ES, Pusic M, Schumacher D, Tolsgaard MG, Tsai CC, Wenghofer E, Touchie C. Data sharing and big data in health professions education: Ottawa consensus statement and recommendations for scholarship. MEDICAL TEACHER 2024; 46:471-485. [PMID: 38306211 DOI: 10.1080/0142159x.2023.2298762] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Accepted: 12/20/2023] [Indexed: 02/04/2024]
Abstract
Changes in digital technology, increasing volume of data collection, and advances in methods have the potential to unleash the value of big data generated through the education of health professionals. Coupled with this potential are legitimate concerns about how data can be used or misused in ways that limit autonomy, equity, or harm stakeholders. This consensus statement is intended to address these issues by foregrounding the ethical imperatives for engaging with big data as well as the potential risks and challenges. Recognizing the wide and ever evolving scope of big data scholarship, we focus on foundational issues for framing and engaging in research. We ground our recommendations in the context of big data created through data sharing across and within the stages of the continuum of the education and training of health professionals. Ultimately, the goal of this statement is to support a culture of trust and quality for big data research to deliver on its promises for health professions education (HPE) and the health of society. Based on expert consensus and review of the literature, we report 19 recommendations in (1) framing scholarship and research through research, (2) considering unique ethical practices, (3) governance of data sharing collaborations that engage stakeholders, (4) data sharing processes best practices, (5) the importance of knowledge translation, and (6) advancing the quality of scholarship through multidisciplinary collaboration. The recommendations were modified and refined based on feedback from the 2022 Ottawa Conference attendees and subsequent public engagement. Adoption of these recommendations can help HPE scholars share data ethically and engage in high impact big data scholarship, which in turn can help the field meet the ultimate goal: high-quality education that leads to high-quality healthcare.
Collapse
Affiliation(s)
| | - Lawrence Grierson
- Department of Family Medicine, McMaster University, Hamilton, Canada
| | - Cassandra Barber
- School of Health Professions Education, Maastricht University, Maastricht, Netherlands
| | - Saad Chahine
- Faculty of Education, Queen's University, Kingston, Canada
| | - Fremen Chichen Chou
- Faculty of Education, Center for Faculty Development, China Medical University Hospital, Taichung City, Taiwan
| | - Jennifer Cleland
- Director of Medical Education Research & Scholarship Unit, Lee Kong Chian School of Medicine, Singapore
| | | | - Eric S Holmboe
- Accreditation Council for Graduate Medical Education, Chicago, IL, USA
| | | | - Daniel Schumacher
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Martin G Tolsgaard
- Copenhagen Academy for Medical Education and Simulation, University of Copenhagen, Copenhagen, Denmark
| | - Chin-Chung Tsai
- Program of Learning Sciences, National Taiwan Normal University, Taipei, Taiwan
| | - Elizabeth Wenghofer
- School of Kinesiology and Health Sciences, Laurentian University, Sudbury, Canada
| | - Claire Touchie
- University of Ottawa/The Ottawa Hospital, Ottawa, Canada
| |
Collapse
|
7
|
Brown A, La J, Keri MI, Hillis C, Razack S, Korah N, Karpinski J, Frank JR, Wong B, Goldman J. In EPAs we trust, is quality and safety a must? A cross-specialty analysis of entrustable professional activity guides. MEDICAL TEACHER 2024:1-9. [PMID: 38527417 DOI: 10.1080/0142159x.2024.2323177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 02/21/2024] [Indexed: 03/27/2024]
Abstract
PURPOSE The inclusion of quality improvement (QI) and patient safety (PS) into CanMEDS reflects an expectation that graduating physicians are competent in these areas upon training completion. To ensure that Canadian postgraduate specialty training achieves this, the translation of QI/PS competencies into training standards as part of the implementation of competency-based medical education requires special attention. METHODS We conducted a cross-specialty, multi-method analysis to examine how QI/PS was incorporated into the EPA Guides across 11 postgraduate specialties in Canada. RESULTS We identify cross-specialty variability in how QI/PS is incorporated, positioned, and emphasized in EPAs and milestones. QI/PS was primarily referenced alongside clinical activities rather than as a sole competency or discrete activity. Patterns were characterized in how QI/PS became incorporated into milestones through repetition and customization. QI/PS was also decoupled, conceptualized, and emphasized differently across specialties. CONCLUSIONS Variability in the inclusion of QI/PS in EPAs and milestones has important implications considering the visibility and influence of EPA Guides in practice. As specialties revisit and revise EPA Guides, there is a need to balance the standardization of foundational QI/PS concepts to foster shared understanding while simultaneously ensuring context-sensitive applications across specialties. Beyond QI/PS, this study illuminates the challenges and opportunities that lie in bridging theoretical frameworks with practical implementation in medical education, prompting broader consideration of how intrinsic roles and emergent areas are effectively incorporated into competency-based medical education.
Collapse
Affiliation(s)
- Allison Brown
- Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Julie La
- Graduate Program in Health Quality, Queen's University, Kingston, Canada
- Department of Surgery, Queen's University, Kingston, Canada
| | | | - Chris Hillis
- Faculty of Health Sciences, McMaster University, Hamilton, Canada
| | - Saleem Razack
- Faculty of Medicine, University of British Columbia, Kelowna, Canada
| | - Nadine Korah
- Faculty of Medicine, McGill University, Montreal, Canada
| | | | - Jason R Frank
- Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Brian Wong
- Centre for Quality Improvement and Patient Safety, Toronto, Canada
- Faculty of Medicine, University of Toronto Temerty, Toronto, Canada
| | - Joanne Goldman
- Centre for Quality Improvement and Patient Safety, Toronto, Canada
- Faculty of Medicine, University of Toronto Temerty, Toronto, Canada
| |
Collapse
|
8
|
Caretta-Weyer HA, Schumacher DJ, Kinnear B. Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2024; 99:243-246. [PMID: 38011041 DOI: 10.1097/acm.0000000000005578] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
ABSTRACT In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.
Collapse
|
9
|
Karnik R, Robinson B, Glickstein J, Schwartz A, Mink R, Neal AE, Frank LH, McMahon CJ, Kim ME, Turner DA, Srivastava S. Minimum Supervision Levels Required for Pediatric Cardiology Fellowship Graduation by Pediatric Cardiology Fellowship Program Directors. Pediatr Cardiol 2023:10.1007/s00246-023-03371-y. [PMID: 38117291 DOI: 10.1007/s00246-023-03371-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Accepted: 11/30/2023] [Indexed: 12/21/2023]
Abstract
Entrustable professional activities (EPAs) are "observable essential tasks expected to be performed by a physician for safe patient care in practice." Six Pediatric Cardiology (PC) EPAs and their level of supervision (LOS) scales were developed by medical educators in PC using a modified Delphi process and reviewed by the Subspecialty Pediatrics Investigator Network (SPIN). However, their general use in assessment for PC fellows for graduation requirements has yet to be studied. The objective of this study was to determine the minimum LOS required for PC fellows to graduate and compare it with the minimum LOS expected for safe and effective practice for the six PC EPAs, from the perspective of the PC Fellowship Program Directors(FPD). All Fellowship Program Directors(FPD) of ACGME-accredited PC fellowships were surveyed through SPIN between April 2017 and August 2017. For each of the PC EPAs, the FPDs were asked to indicate the minimum LOS expected for graduation and whether they would allow a fellow to graduate if this level was not achieved and the minimum LOS expected for a practicing pediatric cardiologist to provide safe and effective patient care. The minimum LOS was defined as the LOS for which no more than 20% of FPDs would want a lower level. The survey response rate was 80% (47/59). The majority of the FPDs did not require a minimum LOS of five corresponding to unsupervised practice in any of the six PC EPAs at graduation. For EPAs related to imaging, arrhythmia management, and management of cardiac problems, the minimum LOS for graduation was 3, corresponding to being "trusted to perform a task with indirect supervision for most simple and a few complex cases." For the EPAs related to interventional cardiology, heart failure pulmonary hypertension, and cardiac intensive care, the minimum LOS for graduation was 2, corresponding to being "trusted to perform a task only with direct supervision and coaching." The minimum LOS considered necessary for safe and effective practice for all but one EPA was 3. For the EPA related to the management of cardiac problems, the minimum LOS for safe practice was 4, corresponding to being "trusted to execute tasks independently except for few complex and critical cases." Most PC FPDs reported they would not require fellows to achieve the highest entrustment level for any of the six PC EPAs for graduation. It is crucial that educational programs evolve to address these essential activities during training better and that stakeholders ensure that graduating PC fellows have adequate resources and infrastructure to continue professional development as early career pediatric cardiologists.
Collapse
Affiliation(s)
- Ruchika Karnik
- Department of Pediatrics, Section of Pediatric Cardiology, Yale School of Medicine, New Haven, CT, USA.
| | - Bradley Robinson
- Department of Pediatrics, Section of Pediatric Cardiology, Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA, USA
| | - Julie Glickstein
- Department of Pediatrics, Division of Pediatric Cardiology, Morgan Stanley Children's Hospital, Columbia University Medical Center, New York, NY, USA
| | - Alan Schwartz
- Department of Medical Education and Department of Pediatrics, University of Illinois at Chicago College of Medicine, Chicago, IL, USA
| | - Richard Mink
- David Geffen School of Medicine at UCLA and The Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, Torrance, CA, USA
| | - Ashley E Neal
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, Heart Institute, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Lowell H Frank
- Division of Cardiology, Children's National Hospital, Washington, DC, USA
| | - Colin J McMahon
- Department of Pediatric Cardiology, Children's Health Ireland at Crumlin, Dublin 12, Ireland
| | - Michael E Kim
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, ON, Canada
| | - David A Turner
- Competency-Based Medical Education, American Board of Pediatrics, Chapel Hill, NC, USA
| | - Shubhika Srivastava
- Department of Pediatrics, Section of Pediatric Cardiology, Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA, USA
| |
Collapse
|
10
|
Poeppelman RS, Moore-Clingenpeel M, Siems A, Mitchell DL, Jani P, Stewart C. Faculty Decision Making in Ad Hoc Entrustment of Pediatric Critical Care Fellows: A National Case-Based Survey. TEACHING AND LEARNING IN MEDICINE 2023:1-8. [PMID: 37933862 DOI: 10.1080/10401334.2023.2269402] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Accepted: 10/02/2023] [Indexed: 11/08/2023]
Abstract
Phenomenon: Ad hoc entrustment decisions reflect a clinical supervisor's estimation of the amount of supervision a trainee needs to successfully complete a task in the moment. These decisions have important consequences for patient safety, trainee learning, and preparation for independent practice. Determinants of these decisions have previously been described but have not been well described for acute care contexts such as critical care and emergency medicine. The ad hoc entrustment of trainees caring for vulnerable patient populations is a high-stakes decision that may differ from other contexts. Critically ill patients and children are vulnerable patient populations, making the ad hoc entrustment of a pediatric critical care medicine (PCCM) fellow a particularly high-stakes decision. This study sought to characterize how ad hoc entrustment decisions are made for PCCM fellows through faculty ratings of vignettes. The authors investigated how acuity, relationship, training level, and task interact to influence ad hoc entrustment decisions. Approach: A survey containing 16 vignettes that varied by four traits (acuity, relationship, training level, and task) was distributed to U.S. faculty of pediatric critical care fellowships in 2020. Respondents determined an entrustment level for each case and provided demographic data. Entrustment ratings were dichotomized by "high entrustment" versus "low entrustment" (direct supervision or observation only). The authors used logistic regression to evaluate the individual and interactive effects of the four traits on dichotomized entrustment ratings. Findings: One hundred seventy-eight respondents from 30 institutions completed the survey (44% institutional response rate). Acuity, relationship, and task all significantly influenced the entrustment level selected but did not interact. Faculty most frequently selected "direct supervision" as the entrustment level for vignettes, including for 24% of vignettes describing fellows in their final year of training. Faculty rated the majority of vignettes (61%) as "low entrustment." There was no relationship between faculty or institutional demographics and the entrustment level selected. Insights: As has been found in summative entrustment for pediatrics, internal medicine, and surgery trainees, PCCM fellows often rated at or below the "direct supervision" level of ad hoc entrustment. This may relate to declining opportunities to practice procedures, a culture of low trust propensity among the specialty, and/or variation in interpretation of entrustment scales.
Collapse
Affiliation(s)
- Rachel Stork Poeppelman
- Department of Pediatrics, Nationwide Children's Hospital, The Ohio State University, Columbus, Ohio, USA
| | - Melissa Moore-Clingenpeel
- Department of Pediatrics, Nationwide Children's Hospital, The Ohio State University, Columbus, Ohio, USA
| | - Ashley Siems
- Department of Pediatrics, Children's National, Washington, DC, USA
| | - Diana L Mitchell
- Department of Pediatrics, Advocate Children's Hospital Park Ridge, Park Ridge, Illinois, USA
- Department of Pediatrics, University of Chicago, Chicago, Illinois, USA
| | - Priti Jani
- Department of Pediatrics, University of Chicago, Chicago, Illinois, USA
| | - Claire Stewart
- Department of Pediatrics, Nationwide Children's Hospital, The Ohio State University, Columbus, Ohio, USA
| |
Collapse
|
11
|
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DEJ, McGann KA, Kesselheim J, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Sauer C, Pitts S, Myers AL, Mahan JD, Dammann CEL, Aye T, Schwartz A. Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment. BMC MEDICAL EDUCATION 2023; 23:720. [PMID: 37789289 PMCID: PMC10548580 DOI: 10.1186/s12909-023-04689-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 09/15/2023] [Indexed: 10/05/2023]
Abstract
BACKGROUND Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment. OBJECTIVES We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME). METHODS In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale. RESULTS One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05). CONCLUSIONS We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.
Collapse
Affiliation(s)
- Richard B Mink
- Department of Pediatrics, David Geffen School of Medicine at UCLA and the Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, 1124 West Carson Street, Torrance, CA, 90502, USA.
| | | | - Bruce E Herman
- University of Utah School of Medicine, Salt Lake, UT, USA
| | - Pnina Weiss
- Department of Pediatrics, Yale School of Medicine, New Haven, CT, USA
| | | | - Diane E J Stafford
- Division of Endocrinology, Department of Pediatrics, Stanford University School of Medicine, Palo Alto, CA, USA
| | - Kathleen A McGann
- Department of Pediatrics, Duke University Medical Center, Durham, NC, USA
| | - Jennifer Kesselheim
- Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Boston, MA, USA
| | | | - Pamela C High
- Alpert Medical School of Brown University, Providence, RI, USA
- Developmental-Behavioral Pediatrics, Hasbro Children's Hospital, Providence, RI, USA
| | - Jill J Fussell
- University of Arkansas for Medical Sciences and Arkansas Children's Hospital, Little Rock, AR, USA
| | - Megan L Curran
- Department of Pediatrics, University of Colorado School of Medicine, Aurora, CO, USA
| | | | - Cary Sauer
- Department of Pediatrics, Emory University School of Medicine and Children's Healthcare of Atlanta, Atlanta, GA, USA
| | - Sarah Pitts
- Division of Adolescent/Young Adult Medicine, Boston Children's Hospital, Boston, MA, USA
| | - Angela L Myers
- Center for Wellbeing, Children's Mercy Hospital and University of Missouri-Kansas City School of Medicine, Kansas City, MO, USA
| | - John D Mahan
- Department of Pediatrics, Nationwide Children's Hospital and The Ohio State University College of Medicine, Columbus, OH, USA
| | | | - Tandy Aye
- Division of Endocrinology, Department of Pediatrics, Stanford University School of Medicine, Palo Alto, CA, USA
| | - Alan Schwartz
- University of Illinois College of Medicine at Chicago, Chicago, IL, USA
| |
Collapse
|
12
|
Brasel KJ, Lindeman B, Jones A, Sarosi GA, Minter R, Klingensmith ME, Whiting J, Borgstrom D, Buyske J, Mellinger JD. Implementation of Entrustable Professional Activities in General Surgery: Results of a National Pilot Study. Ann Surg 2023; 278:578-586. [PMID: 37436883 DOI: 10.1097/sla.0000000000005991] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/14/2023]
Abstract
OBJECTIVE The ongoing complexity of general surgery training has led to an increased focus on ensuring the competence of graduating residents. Entrustable professional activities (EPAs) are units of professional practice that provide an assessment framework to drive competency-based education. The American Board of Surgery convened a group from the American College of Surgeons, Accreditation Council for Graduate Medical Education (ACGME) Surgery Review Committee, and Association of Program Directors in Surgery to develop and implement EPAs in a pilot group of residency programs across the country. The objective of this pilot study was to determine the feasibility and utility of EPAs in general surgery resident training. METHODS 5 EPAs were chosen based on the most common procedures reported in ACGME case logs and by practicing general surgeons (right lower quadrant pain, biliary disease, inguinal hernia), along with common activities covering additional ACGME milestones (performing a consult, care of a trauma patient). Levels of entrustment assigned (1 to 5) were observation only, direct supervision, indirect supervision, unsupervised, and teaching others. Participating in site recruitment and faculty development occurred from 2017 to 2018. EPA implementation at individual residency programs began on July 1, 2018, and was completed on June 30, 2020. Each site was assigned 2 EPAs to implement and collected EPA microassessments on residents for those EPAs. The site clinical competency committees (CCC) used these microassessments to make summative entrustment decisions. Data submitted to the independent deidentified data repository every 6 months included the number of microassessments collected per resident per EPA and CCC summative entrustment decisions. RESULTS Twenty-eight sites were selected to participate in the program and represented geographic and size variability, community, and university-based programs. Over the course of the 2-year pilot programs reported on 14 to 180 residents. Overall, 6,272 formative microassessments were collected (range, 0 to 1144 per site). Each resident had between 0 and 184 microassessments. The mean number of microassessments per resident was 5.6 (SD = 13.4) with a median of 1 [interquartile range (IQR) = 6]. There were 1,763 summative entrustment ratings assigned to 497 unique residents. The average number of observations for entrustment was 3.24 (SD 3.61) with a median of 2 (IQR 3). In general, PGY1 residents were entrusted at the level of direct supervision and PGY5 residents were entrusted at unsupervised practice or teaching others. For each EPA other than the consult EPA, the degree of entrustment reported by the CCC increased by resident level. CONCLUSIONS These data provide evidence that widespread implementation of EPAs across general surgery programs is possible, but variable. They provide meaningful data that graduating chief residents are entrusted by their faculty to perform without supervision for several common general surgical procedures and highlight areas to target for the successful widespread implementation of EPAs.
Collapse
Affiliation(s)
- Karen J Brasel
- Department of Surgery, Oregon Health and Science University, Portland, OR
| | - Brenessa Lindeman
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL
| | | | - George A Sarosi
- Department of Surgery, University of Florida, Gainesville, FL
| | - Rebecca Minter
- Department of Surgery, University of Wisconsin, Madison, WI
| | - Mary E Klingensmith
- Accreditation Council for Graduate Medical Education, Chicago, IL
- Department of Surgery, Washington University in St. Louis, St. Louis, MO
| | - James Whiting
- Department of Surgery, Maine Medical Center, Portland, ME
| | - David Borgstrom
- Department of Surgery, University of West Virginia, Morgantown, WV
| | - Jo Buyske
- American Board of Surgery, Philadelphia, PA
| | | |
Collapse
|
13
|
Kinnear B, Santen SA, Kelleher M, Martini A, Ferris S, Edje L, Warm EJ, Schumacher DJ. How Does TIMELESS Training Impact Resident Motivation for Learning, Assessment, and Feedback? Evaluating a Competency-Based Time-Variable Training Pilot. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:828-835. [PMID: 36656286 DOI: 10.1097/acm.0000000000005147] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
PURPOSE As competency-based medical education has become the predominant graduate medical education training model, interest in time-variable training has grown. Despite multiple competency-based time-variable training (CBTVT) pilots ongoing in the United States, little is known about how this training approach impacts learners. The authors aim to explore how their CBTVT pilot program impacted resident motivation for learning, assessment, and feedback. METHOD The authors performed a qualitative educational case study on the Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) program at the University of Cincinnati from October 2020 through March 2022. Semistructured interviews were conducted with TIMELESS residents (n = 9) approximately every 6 months to capture experiences over time. The authors used inductive thematic analysis to develop themes and compared their findings with existing theories of learner motivation. RESULTS The authors developed 2 themes: TIMELESS had variable effects on residents' motivation for learning and TIMELESS increased resident engagement with and awareness of the program of assessment. Participants reported increased motivation to learn and seek assessment, though some felt a tension between performance (e.g., advancement through the residency program) and growth (e.g., improvement as a physician). Participants became more aware of the quality of assessments they received, in part due to TIMELESS increasing the perceived stakes of assessment, and reported being more deliberate when assessing other residents. CONCLUSIONS Resident motivation for learning, assessment, and feedback was impacted in ways that the authors contextualize using current theories of learner motivation (i.e., goal orientation theory and attribution theory). Future research should investigate how interventions, such as coaching, guided learner reflection, or various CBTVT implementation strategies, can help keep learners oriented toward mastery learning rather than toward performance.
Collapse
Affiliation(s)
- Benjamin Kinnear
- B. Kinnear is associate professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-0052-4130
| | - Sally A Santen
- S.A. Santen is professor of emergency medicine, Department of Emergency Medicine, Virginia Commonwealth University School of Medicine, Richmond, Virginia, and University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-8327-8002
| | - Matthew Kelleher
- M. Kelleher is assistant professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6400-1745
| | - Abigail Martini
- A. Martini is a clinical research coordinator with emergency medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
| | - Sarah Ferris
- S. Ferris is a research administrator, Clinical Trials Unit, Michigan Medicine Research, University of Michigan, Ann Arbor, Michigan
| | - Louito Edje
- L. Edje is professor of family and community medicine, Departments of Medical Education and of Family and Community Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Eric J Warm
- E.J. Warm is professor of internal medicine and program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6088-2434
| | - Daniel J Schumacher
- D.J. Schumacher is professor of pediatrics, Cincinnati Children's Hospital Medical Center and Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| |
Collapse
|
14
|
Allen K, Najjar M, Ostermeier A, Washington N, Genies MC, Hazle M, Hardy C, Lewis K, McDaniel L, McFarlane DJ, Macias C, Molloy MJ, Perry MF, Piper L, Sevov C, Titus L, Toth H, Unaka NI, Weisgerber MC, Kasick R. The Autonomy Toolbox: A Multicenter Collaborative to Promote Resident Autonomy. Hosp Pediatr 2023; 13:490-503. [PMID: 37153964 DOI: 10.1542/hpeds.2022-006827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
OBJECTIVES Autonomy is necessary for resident professional development and well-being. A recent focus on patient safety has increased supervision and decreased trainee autonomy. Few validated interventions exist to improve resident autonomy. We aimed to use quality improvement methods to increase our autonomy metric, the Resident Autonomy Score (RAS), by 25% within 1 year and sustain for 6 months. METHODS We developed a bundled-intervention approach to improve senior resident (SR) perception of autonomy on Pediatric Hospital Medicine (PHM) services at 5 academic children's hospitals. We surveyed SR and PHM faculty perceptions of autonomy and targeted interventions toward areas with the highest discordance. Interventions included SR and faculty development, expectation-setting huddles, and SR independent rounding. We developed a Resident Autonomy Score (RAS) index to track SR perceptions over time. RESULTS Forty-six percent of SRs and 59% of PHM faculty completed the needs assessment survey querying how often SRs were afforded opportunities to provide autonomous medical care. Faculty and SR ratings were discordant in these domains: SR input in medical decisions, SR autonomous decision-making in straightforward cases, follow-through on SR plans, faculty feedback, SR as team leader, and level of attending oversight. The RAS increased by 19% (3.67 to 4.36) 1 month after SR and faculty professional development and before expectation-setting and independent rounding. This increase was sustained throughout the 18-month study period. CONCLUSIONS SRs and faculty perceive discordant levels of SR autonomy. We created an adaptable autonomy toolbox that led to sustained improvement in perception of SR autonomy.
Collapse
Affiliation(s)
- Karen Allen
- Division of Hospital Medicine, Department of Pediatrics
| | | | - Austen Ostermeier
- Division of Hospital Medicine, Department of Pediatrics, Washington University in St Louis School of Medicine, St Louis, Missouri
| | - Nicole Washington
- Section of Hospital Medicine, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
- Department of Pediatrics, The University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania
| | - Marquita C Genies
- Division of Hospital Medicine, Department of Pediatrics, John Hopkins University School of Medicine, Baltimore, Maryland
| | - Matthew Hazle
- Division of Hospital Medicine, Department of Pediatrics, University of Michigan, Ann Arbor, Michigan
| | | | - Kristen Lewis
- Division of Hospital Medicine, Department of Pediatrics
- Division of Hospital Medicine, Department of Internal Medicine, Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Lauren McDaniel
- Division of Hospital Medicine, Department of Pediatrics, Seattle Children's Hospital, Seattle, Washington
| | - Daniel J McFarlane
- Division of Hospital Medicine, Department of Pediatrics
- Division of Hospital Medicine, Department of Internal Medicine, Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Charlie Macias
- Planning and Business Development, Nationwide Children's Hospital, Columbus, Ohio
| | - Matthew J Molloy
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | | | - Laura Piper
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Claire Sevov
- Division of Hospital Medicine, Department of Pediatrics
- Division of Hospital Medicine, Department of Internal Medicine, Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Lauren Titus
- Division of Hospital Medicine, Children's Wisconsin, Milwaukee, Wisconsin
- Departments of Pediatrics
| | - Heather Toth
- Division of Hospital Medicine, Children's Wisconsin, Milwaukee, Wisconsin
- Departments of Pediatrics
- Internal Medicine, Medical College of Wisconsin, Milwaukee, Wisconsin
| | - Ndidi I Unaka
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Michael C Weisgerber
- Division of Hospital Medicine, Children's Wisconsin, Milwaukee, Wisconsin
- Departments of Pediatrics
| | - Rena Kasick
- Division of Hospital Medicine, Department of Pediatrics
| |
Collapse
|
15
|
Goldman MP, Auerbach MA. Autonomy Is Desired, Entrustment Is What Matters. Hosp Pediatr 2023; 13:e150-e152. [PMID: 37153966 DOI: 10.1542/hpeds.2023-007205] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Affiliation(s)
- Michael P Goldman
- Department of Pediatrics, Section of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Marc A Auerbach
- Department of Pediatrics, Section of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut
| |
Collapse
|
16
|
Langhan ML, Stafford DEJ, Myers AL, Herman BE, Curran ML, Czaja AS, Turner DA, Weiss P, Mink R. Clinical competency committee perceptions of entrustable professional activities and their value in assessing fellows: A qualitative study of pediatric subspecialty program directors. MEDICAL TEACHER 2023; 45:650-657. [PMID: 36420760 DOI: 10.1080/0142159x.2022.2147054] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/18/2023]
Abstract
OBJECTIVES To examine the composition and processes of Clinical Competency Committees (CCCs) assigning entrustable professional activity (EPA) levels of supervision for pediatric subspecialty fellows and to examine fellowship program director (FPD) perspectives about using EPAs to determine fellows' graduation readiness. METHODS A qualitative study was performed using one-on-one interviews with a purposeful sample of pediatric subspecialty FPDs to yield a thematic analysis. Semi-structured interview guides were used for participants who self-identified as EPA users or non-users. Inductive analysis and coding were performed on transcripts until theoretical sufficiency was attained. RESULTS Twenty-eight FPDs were interviewed. There was significant variability in the composition and processes of CCCs across subspecialties. FPDs felt that CCCs intuitively understand what entrustment means, allowing for ease of application of level of supervision (LOS) scales and consensus. FPDs perceived that EPAs provided a global assessment of fellows and are one tool to determine graduation readiness. CONCLUSIONS Although there was variability in the makeup and processes of CCCs across subspecialties, FPDs believe EPAs are intuitive and relatively easy to implement. Consensus can be reached easily using EPA-specific LOS scales focusing on entrustment. FPDs desire a better understanding of how EPAs should be used for graduation.
Collapse
Affiliation(s)
- Melissa L Langhan
- Department of Pediatrics and Emergency Medicine, Section of Emergency Medicine, Yale University School of Medicine, New Haven, CT, USA
| | - Diane E J Stafford
- Department of Pediatrics, Division of Endocrinology, Stanford University School of Medicine, Stanford, CA, USA
| | - Angela L Myers
- Department of Pediatrics, Children's Mercy, Kansas City, University of Missouri-Kansas City School of Medicine, Kansas City, MO, USA
| | - Bruce E Herman
- Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Megan L Curran
- Department of Pediatrics, Section of Rheumatology, University of Colorado School of Medicine, Aurora, CO, USA
| | - Angela S Czaja
- Department of Pediatrics, Section of Critical Care, University of Colorado School of Medicine, Aurora, CO, USA
| | | | - Pnina Weiss
- Department of Pediatrics, Section of Pulmonology, Allergy, Immunology and Sleep Medicine, Yale University School of Medicine, New Haven, CT, USA
| | - Richard Mink
- David Geffen School of Medicine at UCLA, Los Angeles, CA, USA
- Department of Pediatrics, Harbor-UCLA Medical Center and The Lundquist Institute for Biomedical Innovation, Torrance, CA, USA
| |
Collapse
|
17
|
Havens JF, Marr MC. Pediatric Psychiatric Emergency Services-Stasis in Crisis. JAMA 2023; 329:1453-1454. [PMID: 37129666 DOI: 10.1001/jama.2023.3693] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Affiliation(s)
- Jennifer F Havens
- Department of Child and Adolescent Psychiatry, New York University Grossman School of Medicine, New York
| | - Mollie C Marr
- Medical Scientist Training Program, Oregon Health & Science University, Portland
| |
Collapse
|
18
|
A Virtual Reality Curriculum to Enhance Residents' Behavioral Health Anticipatory Guidance Skills: A Pilot Trial. Acad Pediatr 2023; 23:185-192. [PMID: 35870801 DOI: 10.1016/j.acap.2022.07.011] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Revised: 06/27/2022] [Accepted: 07/04/2022] [Indexed: 01/19/2023]
Abstract
OBJECTIVE Effective educational strategies to train and assess residents' skills in behavioral health competencies have been limited. We hypothesized that a training curriculum using virtual reality (VR) simulations would enhance residents' evidence-based skills related to behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI). METHODS We conducted a randomized controlled pilot trial of our curriculum within an urban academic clinic. Intervention and control groups received four 15-minute, monthly didactics on BHAG and MI followed by VR simulations to 1) deliberately practice delivering BHAG by verbally counseling avatars, who responded real-time (intervention group) or 2) assessing pediatric respiratory distress (control group). Two months post-VR training, all residents participated in a unique VR behavioral health scenario which was recorded and coded by pediatric psychologists via an observation instrument to assess residents' skills. Differences in BHAG and MI competencies were assessed via independent samples t tests for continuous variables and Fisher's exact tests for categorical variables. RESULTS Sixty-six percent (n = 55) of eligible residents completed all study protocols comprising 35 intervention and 20 control participants. Intervention participants demonstrated an increase in BHAG skills (P = .002), percentage of open-ended questions asked (P = .04), and percentage of MI adherent behaviors (P = .04) when compared to the control group. CONCLUSIONS VR may be an effective educational strategy for residents to acquire BHAG and MI skills though deliberate practice. Next steps will focus on assessing the generalizability of the VR curriculum as well as exploring the opportunities to enhance its scalability through artificial intelligence.
Collapse
|
19
|
Roth LT, Cooper MB, Lurie B, Wueste B. Developing an Entrustable Professional Activity to Improve the Care of LGBTQ+ Youth. Acad Pediatr 2022; 23:697-702. [PMID: 36122833 DOI: 10.1016/j.acap.2022.09.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/18/2022] [Revised: 09/08/2022] [Accepted: 09/10/2022] [Indexed: 11/01/2022]
Abstract
Despite the multitude of health inequities often experienced by lesbian, gay, bisexual, transgender, and queer (LGBTQ)+ youth, few training programs consistently and comprehensively address these disparities through education and assessment of learners. Over the past decade, programs have begun to incrementally increase the number of hours dedicated to LGBTQ+ health training; however, the lack of standardization for curricula and assessment remains a significant barrier. As medical education shifts to a competency-based approach, focusing on patient outcomes and the competencies trainees must achieve to meet those outcomes, we must take this opportunity to develop curricula and assessment tools that emphasize inclusive, affirming, equitable, and culturally humble care for LGBTQ+ youth. Entrustable professional activities (EPA) are increasingly used to educate and assess trainees on their progression from full supervision to unsupervised practice. Given their multifaceted use to standardize curricula, assess learners, and highlight programmatic gaps in education, we believe an EPA is the ideal tool to systematically improve LGBTQ+ health training in graduate medical education. Our team of national experts in this field developed an LGBTQ+ health EPA. We describe our methods of development, the functions and curricular components of the EPA, and the many applications for widespread program use.
Collapse
Affiliation(s)
- Lauren T Roth
- Department of Pediatrics (LT Roth), Children's Hospital at Montefiore/Albert Einstein College of Medicine, Bronx, NY.
| | - M Brett Cooper
- Department of Pediatrics (MB Cooper), UT Southwestern Medical Center/Children's Medical Center Dallas
| | - Brian Lurie
- Department of Pediatrics (B Lurie), Atrium Health/Levine Children's Hospital, NC
| | - Beth Wueste
- Department of Pediatrics (B Wueste), UT Health San Antonio, Tex
| |
Collapse
|
20
|
Brown DR, Moeller JJ, Grbic D, Andriole DA, Cutrer WB, Obeso VT, Hormann MD, Amiel JM. Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020. JAMA Netw Open 2022; 5:e2233342. [PMID: 36156144 PMCID: PMC9513644 DOI: 10.1001/jamanetworkopen.2022.33342] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
IMPORTANCE Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency. OBJECTIVE To assess progress in developing an entrustment process in the Core EPAs framework. DESIGN, SETTING, AND PARTICIPANTS In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database. INTERVENTIONS Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations. MAIN OUTCOMES AND MEASURES On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded. RESULTS Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs. CONCLUSIONS AND RELEVANCE These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.
Collapse
Affiliation(s)
- David R. Brown
- Division of Family and Community Medicine, Department of Humanities, Health, and Society, Florida International University Herbert Wertheim College of Medicine, Miami
| | - Jeremy J. Moeller
- Department of Neurology, Yale University School of Medicine, New Haven, Connecticut
| | - Douglas Grbic
- Medical Education Research, Association of American Medical Colleges, Washington, District of Columbia
| | - Dorothy A. Andriole
- Medical Education Research, Association of American Medical Colleges, Washington, District of Columbia
| | - William B. Cutrer
- Department of Pediatrics, Division of Critical Care Medicine at Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Vivian T. Obeso
- Division of Internal Medicine, Department of Translational Medicine, Florida International University Herbert Wertheim College of Medicine, Miami
| | - Mark D. Hormann
- Division of Community and General Pediatrics, Department of Pediatrics, McGovern Medical School at UTHealth, Houston, Texas
| | - Jonathan M. Amiel
- Dean’s Office, Columbia University Vagelos College of Physicians and Surgeons, New York, New York
- Department of Psychiatry, Columbia University Vagelos College of Physicians and Surgeons, New York, New York
| |
Collapse
|
21
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Elshoff E, Kinnear B. Making prospective entrustment decisions: Knowing limits, seeking help and defaulting. MEDICAL EDUCATION 2022; 56:892-900. [PMID: 35263474 DOI: 10.1111/medu.14797] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 02/28/2022] [Accepted: 03/02/2022] [Indexed: 06/14/2023]
Abstract
INTRODUCTION Many studies focused on entrustment have not considered prospective entrustment decisions, where clinical competency and entrustment committees determine how much supervision trainees will require in the future for tasks occurring in not completely known contexts. The authors sought to explore factors that influence and determine prospective entrustment decisions made by members of such committees in graduate medical education (GME) and undergraduate medical education (UME). METHODS The authors conducted a constructivist grounded theory study with 23 faculty participants from GME and UME clinical competency and entrustment committees in the United States between October 2020 and March 2021. Interviews sought to explore factors and considerations participants weigh in making prospective entrustment decisions about trainees. Data collection and analysis occurred in an iterative fashion, ensuring constant comparison. Theoretical sampling was used to confirm, disconfirm and elaborate on the evolving results. RESULTS Trainees' ability to know limits and seek help is the foundation of participants' prospective entrustment decision making. Most participants, however, describe a presumption of trainee readiness to progress and describe commonly making default prospective entrustment decisions unless 'red flags' in performance are present. Although participants desire sufficient and trusted data about trainee performance to inform decisions, they often lack it. Finally, the perceived permanence and consequences of prospective entrustment decisions influence how participants weigh other factors contributing to prospective entrustment decisions. CONCLUSION Trainees' ability to know limits and seek help appears to be the foundation of prospective entrustment decision making. Training programmes should strive to collect and employ robust data supporting and questioning the presence of these attributes.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Boston Medical Center, Boston University School of Medicine, Boston, Massachusetts, USA
| | - Ariel S Winn
- Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | | | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
22
|
Hsu DC, Baghdassarian A, Caglar D, Rose JA, Herman BE, Schwartz A, Mink R, Langhan ML. Pediatric Emergency Medicine Fellowship Program Directors' Viewpoint: Minimum Levels of Entrustment for Graduating Fellows and Practicing Physicians to Perform the Subspecialty's Professional Activities. Pediatr Emerg Care 2022:00006565-990000000-00110. [PMID: 35947053 DOI: 10.1097/pec.0000000000002817] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVES Entrustable Professional Activities (EPAs) are essential tasks physicians perform within their professions. Entrustment levels that pediatric emergency medicine (PEM) fellowship program directors (FPDs) expect graduating fellows to achieve for PEM-specific and common pediatric subspecialty EPAs remain unreported. This study aims to determine minimum entrustment levels FPDs require fellows to achieve to graduate from fellowship and to compare FPD expectations for fellows versus practicing PEM physicians. METHODS Secondary analysis of PEM-specific data from a national multispecialty cross-sectional survey of pediatric subspecialty FPDs. For 6 PEM-specific and 7 common pediatric subspecialty EPAs, PEM FPDs indicated (1) minimum entrustment levels fellows should achieve by training completion, (2) whether they would allow a fellow to graduate below these minimum levels, and (3) minimum levels for safe and effective practice by PEM physicians. Minimum levels were defined as the level that more than 80% of FPDs would not drop below. RESULTS Sixty of 77 PEM FPDs (78%) completed the survey. Most respondents did not require fellows to achieve the highest level (level 5-no supervision) by graduation for any PEM-specific EPAs. The median level FPDs expected for practicing PEM physicians was 5 (trusted to perform without supervision) for EPAs 1 and 4 and level 4 (indirect supervision for complex cases) for the remaining PEM-specific EPAs. Minimum levels expected by FPDs for common subspecialty pediatric EPAs were lower for both groups. CONCLUSIONS Most PEM FPDs indicated that they would graduate fellows before their achievement of the highest entrustment level for all EPAs. Most also indicated that they do not expect practicing PEM physicians to perform all EPAs without supervision. These findings indicate need for stakeholders to evaluate current structure and outcomes of PEM fellowship programs and for institutions and organizations to ensure adequate support in time and resources for ongoing learning for practicing PEM physicians.
Collapse
Affiliation(s)
- Deborah C Hsu
- From the Baylor College of Medicine, Texas Children's Hospital, Houston, TX
| | - Aline Baghdassarian
- Inova L.J. Murphy Children's Hospital, Inova Fairfax Medical Campus, UVA School of Medicine, Falls Church, VA
| | - Derya Caglar
- Division of Emergency Medicine, University of Washington, Seattle Children's Hospital, Seattle, WA
| | - Jerri A Rose
- Division of Pediatric Emergency Medicine, Rainbow Babies and Children's Hospital, Case Western Reserve University School of Medicine, Cleveland, OH
| | - Bruce E Herman
- Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT
| | | | - Richard Mink
- Division of Pediatric Critical Care Medicine, Harbor-UCLA Medical Center, The Lundquist Institute for Biomedical Innovation and David Geffen School of Medicine, Los Angeles, CA
| | | |
Collapse
|
23
|
Pitts S, Schwartz A, Carraccio CL, Herman BE, Mahan JD, Sauer CG, Dammann CEL, Aye T, Myers AL, Weiss PG, Turner DA, Hsu DC, Stafford DEJ, Chess PR, Fussell JJ, McGann KA, High P, Curran ML, Mink RB. Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activities Across Subspecialties. Acad Pediatr 2022; 22:881-886. [PMID: 34936942 DOI: 10.1016/j.acap.2021.12.019] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/28/2021] [Revised: 12/13/2021] [Accepted: 12/16/2021] [Indexed: 11/17/2022]
Abstract
OBJECTIVE To determine the relationship between level of supervision (LOS) ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) with their associated subcompetency milestones across subspecialties and by fellowship training year. METHODS Clinical Competency Committees (CCCs) in 14 pediatric subspecialties submitted LOS ratings for 6 Common Subspecialty EPAs and subcompetency milestone levels mapped to these EPAs. We examined associations between these subcompetency milestone levels and LOS ratings across subspecialty training year by fitting per-EPA linear mixed effects models, regressing LOS rating on milestone level and on training year. RESULTS CCCs from 211 pediatric fellowship programs provided data for 369 first, 336 second, and 331 third year fellows. Mean subcompetency milestone levels increased similarly among subspecialties for most EPAs compared with the reference, Adolescent Medicine. Mean subcompetency milestones mapped to each EPA and mean EPA LOS ratings generally increased by training year across all subspecialties. CONCLUSIONS Subcompetency milestones levels mapped to each Common Subspecialty EPA and the EPA LOS ratings increase similarly across subspecialties and by training year, providing validity evidence for using EPA LOS to assess pediatric subspecialty trainee performance. This study supports the development of tools to facilitated the CCC evaluation process across all pediatric subspecialties.
Collapse
Affiliation(s)
- Sarah Pitts
- Department of Pediatrics, Harvard Medical School, Boston Children's Hospital (S Pitts), Boston, Mass.
| | - Alan Schwartz
- Departments of Medical Education and Pediatrics, University of Illinois at Chicago College of Medicine (A Schwartz), Chicago, Ill
| | - Carol L Carraccio
- Competency-Based Medical Education, American Board of Pediatrics (CL Carraccio and DA Turner), Chapel Hill, NC
| | - Bruce E Herman
- Department of Pediatrics, University of Utah School of Medicine (BE Herman), Salt Lake City, Utah
| | - John D Mahan
- Department of Pediatrics, The Ohio State University College of Medicine, Nationwide Children's Hospital (JD Mahan), Columbus, Ohio
| | - Cary G Sauer
- Department of Pediatrics, Emory University School of Medicine, Children's Healthcare of Atlanta (CG Sauer), Atlanta, Ga
| | | | - Tandy Aye
- Department of Pediatrics, Stanford University School of Medicine (T Aye), Los Gatos, Calif
| | - Angela L Myers
- Children's Mercy Kansas City, UMKC School of Medicine (AL Myers), Kansas City, Mo
| | - Pnina G Weiss
- Department of Pediatrics, Yale School of Medicine (PG Weiss), New Haven, Conn
| | - David A Turner
- Competency-Based Medical Education, American Board of Pediatrics (DA Turner), Chapel Hill, NC
| | - Deborah C Hsu
- Department of Pediatrics, Baylor College of Medicine/Texas Children's Hospital (DC Hsu), Houston, Tex
| | - Diane E J Stafford
- Department of Pediatrics, Stanford University School of Medicine, Stanford University School of Medicine, Center of Academic Medicine (DEJ Stafford), Stanford, Calif; Division of Pediatric Endocrinology (DEJ Stafford), Palo Alto, Calif
| | - Patricia R Chess
- Department of Pediatrics, University of Rochester School of Medicine and Dentistry (PR Chess), Rochester, NY
| | - Jill J Fussell
- Department of Pediatrics, University of Arkansas for Medical Sciences (JJ Fussell), Little Rock, Ark
| | - Kathleen A McGann
- Department of Pediatrics, Duke University Medical Center (KA McGann), Durham, NC
| | - Pamela High
- Department of Pediatrics, W. Alpert Medical School of Brown University (P High), Providence, RI
| | - Megan L Curran
- Department of Pediatrics, University of Colorado (ML Curran), Aurora, Colo
| | - Richard B Mink
- Department of Pediatrics, David Geffen School of Medicine, University of California, Los Angeles (UCLA) (RB Mink); Lundquist Institute for Biomedical Innovation at Harbor UCLA Medical Center (RB Mink), Torrance, Calif
| |
Collapse
|
24
|
Murphy Salem S, Chase B, Newman LR, Cohen AP, Cheston C, Huth K. Perspectives on Complex Care Training in a Large Academic Pediatric Training Program. Acad Pediatr 2022; 22:867-872. [PMID: 35306188 DOI: 10.1016/j.acap.2022.03.008] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Revised: 01/23/2022] [Accepted: 03/14/2022] [Indexed: 11/01/2022]
Abstract
OBJECTIVE To identify gaps and opportunities in complex care training for pediatric residents. METHODS Residents in an academic pediatric residency program were surveyed about: training experiences in complex care; self-entrustment in key clinical activities in complex care; educational strategies that would increase preparedness; and recommendations for curriculum development. We used descriptive statistics for quantitative data and content analysis for free-text responses. RESULTS Of the 160 residents surveyed, 110 (69%) participated. Most participants reported prior clinical exposure to children with medical complexity (CMC; 106, 96%) during both inpatient (82, 75%) and outpatient (88, 80%) clinical rotations. Mean self-entrustment was at or below "somewhat confident" for all clinical activities in complex care, for residents in all postgraduate years. Clinical activities with highest reported self-entrustment included evaluating aspiration into the airway, nutritional issues, care coordination, and evaluating pain. Lowest self-entrustment was reported for facilitating transition to adult care, managing medical technologies, and safety/emergency planning. In terms of educational strategies, participants recommended inpatient encounters with an expert preceptor teaching about evaluating aspiration, pain/irritability and dysmotility (>50%); discussions with patients/families for advocacy, difficult discussions, and transition to adult care (>40%); and hands-on practice for medical technology care (>40%). CONCLUSIONS Pediatric residents report limited self-entrustment in performing key clinical activities in complex care, including for residents at the end of their last postgraduate year. Future curriculum development should prioritize direct observation of clinical encounters with CMC by expert preceptors, partnership with patients and families of CMC, and hands-on simulation.
Collapse
Affiliation(s)
- Sinead Murphy Salem
- Department of Pediatrics (S Murphy Salem, B Chase, and Kathleen Huth), Boston Children's Hospital, Boston, Mass; Department of Pediatrics (S Murphy Salem, B Chase, and C Cheston), Boston Medical Center, Boston, Mass.
| | - Brenna Chase
- Department of Pediatrics (S Murphy Salem, B Chase, and Kathleen Huth), Boston Children's Hospital, Boston, Mass; Department of Pediatrics (S Murphy Salem, B Chase, and C Cheston), Boston Medical Center, Boston, Mass
| | - Lori R Newman
- Department of Education (LR Newman), Boston Children's Hospital, Boston, Mass
| | - Amy P Cohen
- Department of Health Policy and Management (AP Cohen), Harvard Chan School of Public Health, Boston, Mass
| | - Christine Cheston
- Department of Pediatrics (S Murphy Salem, B Chase, and C Cheston), Boston Medical Center, Boston, Mass
| | - Kathleen Huth
- Department of Pediatrics (S Murphy Salem, B Chase, and Kathleen Huth), Boston Children's Hospital, Boston, Mass
| |
Collapse
|
25
|
Harrington KL, Teramoto M, Black L, Carey H, Hartley G, Yung E, Osborne J, Goldberg P, Tran RT, Hillegass E. Physical Therapist Residency Competency-Based Education: Development of an Assessment Instrument. Phys Ther 2022; 102:6535133. [PMID: 35225348 DOI: 10.1093/ptj/pzac019] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/25/2021] [Revised: 09/20/2021] [Accepted: 01/05/2022] [Indexed: 11/13/2022]
Abstract
OBJECTIVE The objective of this study was to develop generic domains of competence (DoC) with associated competencies and milestones for physical therapist residency education. This work was intended to culminate in establishing validity evidence to support a competency-based assessment instrument that could be used by residency programs to evaluate resident achievement of the competencies, regardless of specialty area. METHODS Employing the modified Delphi method, a residency education work group developed an evaluation instrument that included 7 DoC and 31 associated competencies with 6 milestones as rating scales for each competency. The instrument was distributed to mentors and residents in accredited physical therapist residency programs to establish validity evidence. Evaluations (measured by milestones) and demographics were collected at 3 time points (program entry, midterm, and final). Scores across these time points were compared using Kruskal-Wallis tests. Reliability was assessed with kappa statistics (interrater reliability) and alpha reliability coefficients (internal consistency). Construct validity was examined using confirmatory factor analysis via structural equation modeling. RESULTS Overall, 237 mentors and 228 residents completed 824 evaluations (460 by mentors and 364 resident self-evaluations) across the time points. Scores significantly increased from entry through final time points. The interrater reliability of the associated behaviors ranged from moderate to substantial agreement (κ = 0.417-0.774). The internal consistency was high for all DoC at every time point (α reliability coefficients = .881-.955 for entry, .857-.925 for midterm, and .824-.902 for final). After confirmatory factor analysis with structural equation modeling was performed, a model that included 7 DoC and 20 associated competencies was proposed. CONCLUSIONS The residency assessment instrument developed demonstrates interrater reliability and validity evidence and therefore supports competency-based assessment of resident clinical performance across specialty areas. Additionally, the instrument aligns the physical therapy profession with other professions, such as medicine, dentistry, and pharmacy, that have transitioned to competency-based education. IMPACT This study outlines the benefits of moving to competency-based education for physical therapist residents, using a sound evaluation tool that evaluates residents across specialty areas. The instrument will allow for transition to competency-based education in physical therapist residency education programs.
Collapse
Affiliation(s)
- Kendra L Harrington
- Residency and Fellowship Education, American Physical Therapy Association, Alexandria, Virginia, USA
| | - Masaru Teramoto
- Division of Physical Medicine and Rehabilitation, University of Utah, Salt Lake City, Utah, USA
| | - Lisa Black
- Department of Physical Therapy, Creighton University, Omaha, Nebraska, USA
| | - Helen Carey
- Krannert School of Physical Therapy, University of Indianapolis, Indianapolis, Indiana, USA
| | - Greg Hartley
- Department of Physical Therapy, University of Miami Miller School of Medicine, Coral Gables, Florida, USA
| | - Emmanuel Yung
- Department of Physical Therapy, Sacred Heart University College of Health Professions, Fairfield, Connecticut, USA
| | - Jacqueline Osborne
- Brooks Rehabilitation Institute of Higher Learning, Jacksonville, Florida, USA
| | | | - Rachel Tabak Tran
- Casa Colina Hospital and Centers for Healthcare, Pomona, California, USA
| | - Ellen Hillegass
- Department of Physical Therapy, Mercer University College of Health Professions, Atlanta, Georgia, USA
| |
Collapse
|
26
|
Ryan MS, Khamishon R, Richards A, Perera R, Garber A, Santen SA. A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:552-561. [PMID: 34074896 DOI: 10.1097/acm.0000000000004189] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. METHOD For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. RESULTS Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. CONCLUSIONS Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Rebecca Khamishon
- R. Khamishon is a fourth-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Adam Garber
- A. Garber is associate professor, Department of Internal Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-7296-2896
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
27
|
Ryan MS, Khan AR, Park YS, Chastain C, Phillipi C, Santen SA, Barron BA, Obeso V, Yingling SL. Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:544-551. [PMID: 34192721 PMCID: PMC10182784 DOI: 10.1097/acm.0000000000004222] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
PURPOSE In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. METHOD The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. RESULTS Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. CONCLUSIONS In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean, Clinical Medical Education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Asra R Khan
- A.R. Khan is associate professor, director, Doctoring and Clinical Skills course, and clerkship director, Department of Internal Medicine, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-2306-4643
| | - Yoon Soo Park
- Y.S. Park is director, Health Professions Education Research, and member of the faculty, Harvard Medical School and Massachusetts General Hospital, Boston, Massachusetts; ORCID: https://orcid.org/0000-0001-8583-4335
| | - Cody Chastain
- C. Chastain is assistant professor, Department of Internal Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Carrie Phillipi
- C. Phillipi is professor and vice chair of education, Department of Pediatrics, Oregon Health & Science University, Portland, Oregon
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean, Assessment, Evaluation, and Scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia
| | - Beth A Barron
- B.A. Barron is associate professor and associate director, Simulation, Department of Internal Medicine, Columbia University School of Medicine, New York, New York
| | - Vivian Obeso
- V. Obeso is associate professor and assistant dean, Curriculum and Medical Education, Department of Internal Medicine, Florida International University, Miami, Florida
| | - Sandra L Yingling
- S.L. Yingling is assistant professor and associate dean, Educational Planning and Quality Improvement, Department of Medical Education, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-9072-7590
| |
Collapse
|
28
|
Huth K, Henry D, Cribb Fabersunne C, Coleman CL, Frank B, Schumacher D, Shah N. A Multistakeholder Approach to the Development of Entrustable Professional Activities in Complex Care. Acad Pediatr 2022; 22:184-189. [PMID: 34600114 DOI: 10.1016/j.acap.2021.09.014] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/22/2021] [Revised: 09/01/2021] [Accepted: 09/06/2021] [Indexed: 01/30/2023]
Abstract
BACKGROUND Care of children with medical complexity (CMC) involves unique knowledge, skills, and attitudes. Previous work has identified curricular priorities in complex care for pediatricians yet there are no established competency frameworks to guide curriculum development. We aimed to develop and appraise Entrustable Professional Activities (EPAs) for the care of CMC with multistakeholder involvement. METHODS We recruited complex care practitioners to develop EPAs using a template for elaborating descriptive elements. A team of clinicians, educators, trainees, and family leaders refined EPAs and mapped content to the Accreditation Council for Graduate Medical Education Milestones. We conducted virtual focus groups to assess whether EPAs represented the essential skills of pediatricians caring for CMC. Focus group data were analyzed using content analysis. RESULTS Content experts developed 11 EPAs for the care of CMC describing knowledge, skills, and attitudes required for attaining competency. EPAs were mapped to 21 of the 21 (100%) reporting pediatric milestones. Focus group participants endorsed and refined EPA content. Categories of feedback included clarifying medical knowledge, expanding on interpersonal communication skills, emphasizing systems-based practice, and affirming family partnership. CONCLUSIONS A systematic approach to developing EPAs for the care of CMC provides a guide for curriculum development and assessment in complex care.
Collapse
Affiliation(s)
- Kathleen Huth
- Department of Pediatrics, Boston Children's Hospital (K Huth), Boston, Mass; Harvard Medical School, (K Huth), Boston, Mass.
| | - Duncan Henry
- UCSF Pediatric Residency Program, UCSF Benioff Children's Hospital (D Henry), San Francisco, Calif; Department of Pediatrics, University of California San Francisco (D Henry), San Francisco, Ca
| | - Camila Cribb Fabersunne
- UCSF Benioff Children's Hospital (CC Fabersunne), San Francisco, Ca; Department of Pediatrics, University of California San Francisco (CC Fabersunne)
| | - Cara L Coleman
- Family Voices, Inc. (CL Coleman), Lorton, Va; Virginia Commonwealth University School of Medicine (CL Coleman), Richmond, Va
| | - Brigit Frank
- Family Voices of Wisconsin (B Frank), Madison, Wis
| | - Daniel Schumacher
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center (D Schumacher), Cincinnati, Ohio
| | - Neha Shah
- Hospital Medicine Division, Children's National Hospital (N Shah), Washington, DC; The George Washington University, School of Medicine and Health Sciences (N Shah), Washington, DC
| |
Collapse
|
29
|
Rudloff J, Wurster Ovalle V, Barzman D, Gonzalez-Del-Rey J. Psychiatric Education for Pediatric Emergency Medicine Fellows: Should We be Doing More? Pediatr Emerg Care 2022; 38:e734-e738. [PMID: 34009898 DOI: 10.1097/pec.0000000000002450] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/03/2023]
Abstract
The number of psychiatric encounters is steadily increasing across all pediatric emergency departments. Based on this growing national crisis, we hypothesized there is a lack of robust education in behavioral/mental health emergencies during pediatric emergency medicine (PEM) fellowship. METHODS This was a cross-sectional survey of PEM fellowship directors. The survey was submitted to the Pediatric Emergency Medicine Program Director Survey Committee via REDCap to all 83 fellowship directors. RESULTS Fifty-five (66%) PEM fellowship directors completed the survey from September 2019 to December 2019. Forty-three (78%) programs provide some type of didactic teaching, although only 7 (13%) programs offer more than 5 hours of didactics over 3 years of fellowship. Six (11%) programs offer electronic resources, 13 (24%) offer simulations in psychiatry, and only 4 (7%) programs require any type of psychiatry rotation. Forty-six (84%) programs do not currently use psychiatric emergencies as a competency for evaluating PEM fellows. Program directors felt significantly more confident in faculty than graduating fellows in the management of numerous diagnoses including agitation (P < 0.001), homicidal ideation (P = 0.044), neuroleptic malignant syndrome (P = 0.006), acute dystonia (P < 0.001), and heroin overdose (P = 0.022). Ninety-six percent felt online curricula would be helpful to address this knowledge gap. CONCLUSIONS Despite growing behavioral/mental health needs in the pediatric emergency department, there continues to be opportunity to improve psychiatric education during PEM fellowship. Program directors felt more confident in faculty than graduating fellows with several psychiatric diagnoses. Pediatric emergency medicine fellowship directors expressed a strong interest in online tools to assist in the education of PEM fellows.
Collapse
Affiliation(s)
- James Rudloff
- From the Cincinnati Children's Hospital Medical Center
| | | | | | | |
Collapse
|
30
|
Yun Z, Jing L, Junfei C, Wenjing Z, Jinxiang W, Tong Y, Aijun Z. Entrustable Professional Activities for Chinese Standardized Residency Training in Pediatric Intensive Care Medicine. Front Pediatr 2022; 10:919481. [PMID: 35859946 PMCID: PMC9289143 DOI: 10.3389/fped.2022.919481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Accepted: 06/10/2022] [Indexed: 11/15/2022] Open
Abstract
BACKGROUND Entrustable professional activities (EPAs) were first introduced by Olle ten Cate in 2005. Since then, hundreds of applications in medical research have been reported worldwide. However, few studies discuss the use of EPAs for residency training in pediatric intensive care medicine. We conducted a pilot study of EPA for pediatric intensive care medicine to evaluate the use of EPAs in this subspecialty. MATERIALS AND METHODS A cross-sectional study was implemented in pediatric intensive care medicine standardized residency training at the Qilu Hospital of Shandong University. An electronic survey assessing EPA performance using eight scales composed of 15 categories were distributed among residents and directors. RESULTS A total of 217 director-assessment and 44 residents' self-assessment questionnaires were collected, both demonstrating a rising trend in scores across postgraduate years. There were significant differences in PGY1-vs.-PGY2 and PGY1-vs.-PGY3 director-assessment scores, while there were no differences in PGY2-vs.-PGY3 scores. PGY had a significant effect on the score of each EPA, while position significantly affected the scores of all EPAs except for EPA1 (Admit a patient) and EPA2 (Select and interpret auxiliary examinations). Gender only significantly affected the scores of EPA6 (Report a case), EPA12 (Perform health education), and EPA13 (Inform bad news). CONCLUSION This study indicates that EPA assessments have a certain discriminating capability among different PGYs in Chinese standardized residency training in pediatric intensive care medicine. Postgraduate year, gender, and resident position affected EPA scores to a certain extent. Given the inconsistency between resident-assessed and director-assessed scores, an improved feedback program is needed in the future.
Collapse
Affiliation(s)
- Zhang Yun
- Department of Pediatrics, Qilu Hospital of Shandong University, Jinan, China
| | - Liu Jing
- Department of Biostatistics, School of Public Health, Cheeloo College of Medicine, Shandong University, Jinan, China
| | - Chen Junfei
- Department of Pediatric Surgery, Qilu Hospital of Shandong University, Jinan, China
| | - Zhang Wenjing
- Department of Obstetrics and Gynecology, Qilu Hospital of Shandong University, Jinan, China
| | - Wu Jinxiang
- Department of Pulmonary and Critical Care Medicine, Qilu Hospital of Shandong University, Jinan, China
| | - Yue Tong
- Medical Training Office, Qilu Hospital of Shandong University, Jinan, China
| | - Zhang Aijun
- Department of Pediatrics, Qilu Hospital of Shandong University, Jinan, China
| |
Collapse
|
31
|
Kerth JL, van Treel L, Bosse HM. The Use of Entrustable Professional Activities in Pediatric Postgraduate Medical Education: A Systematic Review. Acad Pediatr 2022; 22:21-28. [PMID: 34256178 DOI: 10.1016/j.acap.2021.07.007] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Revised: 05/28/2021] [Accepted: 07/01/2021] [Indexed: 12/27/2022]
Abstract
BACKGROUND Entrustable Professional Activities (EPAs) provide a framework to make judgments of trainees' abilities in several settings including postgraduate medical education. No systematic review of the role of EPAs in pediatrics has yet been performed. OBJECTIVES In our systematic review, we sought to determine the use of EPAs in pediatrics to identify research gaps, summarize and discuss evidence relating to the development, implementation, and assessment. DATA SOURCES Medline, Scopus, PsycINFO, MedEdPortal, and Web of Science. STUDY ELIGIBILITY CRITERIA Two independent reviewers used a structured screening protocol in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and Association for Medical Education in Europe's guide for systematic reviews to include all articles reporting on EPAs in postgraduate medical education and pediatrics in particular. There were no restrictions due to language, study design, or participants. STUDY APPRAISAL AND SYNTHESIS METHODS Data on development, implementation, feasibility, acceptance, and assessment of EPAs were extracted and analyzed by the 2 independent researchers. RESULTS Twenty-eight articles published between 2014 and 2020 were included in the review. We found an increase in publications and a notable shift from descriptions of development processes toward aspects beyond development, ie, implementation, feasibility, acceptance/perception, and assessment. LIMITATIONS Studies from non-English-speaking countries are scarce which might lead to an inaccurate representation of actual international practice. CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS We provide a comprehensive overview of EPAs in pediatrics to guide future curriculum developers in collaborative development, implementation and assessment of EPAs in pediatric postgraduate medical education.
Collapse
Affiliation(s)
- Janna-Lina Kerth
- Department of General Pediatrics, Pediatric Cardiology, and Neonatology, University Children's Hospital Düsseldorf, Heinrich Heine University, Düsseldorf, Germany.
| | - Lena van Treel
- Department of General Pediatrics, Pediatric Cardiology, and Neonatology, University Children's Hospital Düsseldorf, Heinrich Heine University, Düsseldorf, Germany
| | - Hans Martin Bosse
- Department of General Pediatrics, Pediatric Cardiology, and Neonatology, University Children's Hospital Düsseldorf, Heinrich Heine University, Düsseldorf, Germany
| |
Collapse
|
32
|
Lee GB, Chiu AM. Assessment and Feedback Methods in Competency-Based Medical Education. Ann Allergy Asthma Immunol 2021; 128:256-262. [PMID: 34929390 DOI: 10.1016/j.anai.2021.12.010] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 12/10/2021] [Accepted: 12/13/2021] [Indexed: 11/15/2022]
Abstract
Graduate and continuing medical education programs have transitioned to a competency-based medical education (CBME) model, where abilities necessary for the practice of medicine, called competencies, are defined as learning outcomes. Each competency contains milestones that describe stages of development from novice to master and provide criteria for assessing learners along this continuum. Assessments in CBME are formative and are intended to assist learners in achieving the next stage in mastery rather than render a final evaluation. Although written examinations are commonly used to evaluate learners, assessments measuring the performance of skills, such as direct observation or patient registries, provide stronger evidence of achievement of learning outcomes. Once assessments identify strengths and areas of improvement in a learner, effective feedback is critical in assisting their professional development. Feedback should be routine, timely, non-threatening, specific, and encourage self-assessment. The "Ask-Tell-Ask" feedback method follows this framework by asking for the learner's self-assessment first, telling the observer's own assessment next, and then finally asking for the learner's questions and action plan to address the issues raised. Effective assessment and feedback in competency based medical education assists learners in their continued professional development and aims to provide evidence of their impact on the health status of patients and communities.
Collapse
Affiliation(s)
- Gerald B Lee
- Emory University School of Medicine; Division of Pulmonary, Allergy & Immunology, Cystic Fibrosis, and Sleep, Department of Pediatrics, Atlanta, Georgia
| | - Asriani M Chiu
- Medical College of Wisconsin; Division of Asthma, Allergy and Clinical Immunology, Department of Pediatrics, Milwaukee, Wisconsin.
| |
Collapse
|
33
|
Liu L, Jiang Z, Qi X, Xie A, Wu H, Cheng H, Wang W, Li H. An update on current EPAs in graduate medical education: A scoping review. MEDICAL EDUCATION ONLINE 2021; 26:1981198. [PMID: 34569433 PMCID: PMC8477952 DOI: 10.1080/10872981.2021.1981198] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/16/2021] [Revised: 08/23/2021] [Accepted: 09/11/2021] [Indexed: 05/31/2023]
Abstract
The purpose of this scoping review is to update the recent progress of EPAs research in GME, focusing on the topical concern of EPAs effectiveness, and to provide a reference for medical researchers in countries/regions interested in introducing EPAs. Guided by Arksey and O'Malley's framework regarding scoping reviews, the researchers, in January 2021, conducted a search in five databases to ensure the comprehensiveness of the literature. After the predetermined process, 29 articles in total were included in this study. The most common areas for the implementation and evaluation of EPAs were Surgery (n = 7,24.1%), Pediatric (n = 5,17.2%) and Internal medicine (n = 4,13.8%), a result that shows a relatively large change in the research trend of EPAs in the last two years. Prior to 2018, EPAs research focused on internal medicine, psychiatry, family medicine, and primary care. The articles in the category of EPAs implementation and evaluation had four main themes: (1) validation of EPAs (n = 16,55.2%); (2) describing the experience of implementing EPAs (n = 11,37.9%); (3) examining the factors and barriers that influence the implementation and evaluation of EPAs (n = 6,20.6%); and (4) researching the experiences of faculty, interns, and other relevant personnel in using EPAs. Training programs were the most common EPAs implementation setting (n = 26,89.6%); direct observation and evaluation (n = 12,41.4%), and evaluation by scoring reports (n = 5,17.2%) were the two most common means of assessing physicians' EPA levels; 19 papers (65.5%) used faculty evaluation, and nine of these papers also used self-assessment (31.0%); the most frequently used tools in the evaluation of EPAs were mainly researcher-made instruments (n = 37.9%), assessment form (n = 7,24.1%), and mobile application (n = 6,20.7%). Although EPAs occupy an increasingly important place in international medical education, this study concludes that the implementation and diffusion of EPAs on a larger scale is still difficult.
Collapse
Affiliation(s)
- Lu Liu
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Zhehan Jiang
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Xin Qi
- Peking University First Hospital, Beijing, China
| | - A’Na Xie
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Hongbin Wu
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Huaqin Cheng
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Weimin Wang
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Haichao Li
- Peking University First Hospital, Beijing, China
| |
Collapse
|
34
|
Park YS, Hamstra SJ, Yamazaki K, Holmboe E. Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents. JAMA Netw Open 2021; 4:e2137179. [PMID: 34874406 PMCID: PMC8652607 DOI: 10.1001/jamanetworkopen.2021.37179] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/25/2022] Open
Abstract
IMPORTANCE Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice. OBJECTIVE To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents. DESIGN, SETTING, AND PARTICIPANTS This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019. INTERVENTIONS Longitudinal formative assessment using the Milestones assessment system reported to the ACGME. MAIN OUTCOMES AND MEASURES To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories. RESULTS This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth. CONCLUSIONS AND RELEVANCE These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.
Collapse
Affiliation(s)
- Yoon Soo Park
- Harvard Medical School, Boston, Massachusetts
- Massachusetts General Hospital, Boston
- University of Illinois at Chicago College of Medicine, Chicago
| | - Stanley J. Hamstra
- Accreditation Council for Graduate Medical Education, Chicago, Illinois
- Department of Surgery, University of Toronto, Toronto, Ontario, Canada
- Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Kenji Yamazaki
- Accreditation Council for Graduate Medical Education, Chicago, Illinois
| | - Eric Holmboe
- Accreditation Council for Graduate Medical Education, Chicago, Illinois
- Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| |
Collapse
|
35
|
Chan MK, Dickson G, Keegan DA, Busari JO, Matlow A, Van Aerde J. A tale of two frameworks: charting a path to lifelong learning for physician leaders through CanMEDS and LEADS. Leadersh Health Serv (Bradf Engl) 2021; ahead-of-print. [PMID: 34738770 DOI: 10.1108/lhs-04-2021-0032] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
PURPOSE The purpose of this paper was to determine the complementarity between the Canadian Medical Education Directions for Specialists (CanMEDS) physician competency and LEADS leadership capability frameworks from three perspectives: epistemological, philosophical and pragmatic. Based on those findings, the authors propose how the frameworks collectively layout pathways of lifelong learning for physician leadership. DESIGN/METHODOLOGY/APPROACH Using a qualitative approach combining critical discourse analysis with a modified Delphi, the authors examined "How complementary the CanMEDS and LEADS frameworks are in guiding physician leadership development and practice" with the following sub-questions: What are the similarities and differences between CanMEDS and LEADS from: An epistemological and philosophical perspective? The perspective of guiding physician leadership training and practice? How can CanMEDS and LEADS guide physician leadership development from medical school to retirement? FINDINGS Similarities and differences exist between the two frameworks from philosophical and epistemological perspectives with significant complementarity. Both frameworks are founded on a caring ethos and value physician leadership - CanMEDS (for physicians) and LEADS (physicians as one of many professions) define leadership similarly. The frameworks share beliefs in the function of leadership, embrace a belief in distributed leadership, and although having some philosophical differences, have a shared purpose (preparing for changing health systems). Practically, the frameworks are mutually supportive, addressing leadership action in different contexts and where there is overlap, complement one another in intent and purpose. ORIGINALITY/VALUE To the best of the authors' knowledge, this is the first paper to map the CanMEDS (physician competency) and LEADS (leadership capabilities) frameworks. By determining the complementarity between the two, synergies can be used to influence physician leadership capacity needed for today and the future.
Collapse
Affiliation(s)
- Ming-Ka Chan
- Department of Pediatrics and Child Health, University of Manitoba, Winnipeg, Canada
| | - Graham Dickson
- School of Leadership Studies, Royal Roads University, Victoria, Canada and Canadian Society of Physician Leaders, Ottawa, Canada
| | - David A Keegan
- Department of Family Medicine, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Jamiu O Busari
- Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands and Department of Pediatrics, Horacio Oduber Hospital, Oranjestad, Aruba
| | - Anne Matlow
- Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | | |
Collapse
|
36
|
Weiss PG, Schwartz A, Carraccio CL, Herman BE, Turner DA, Aye T, Fussell JJ, Kesselheim J, Mahan JD, McGann KA, Myers A, Stafford DEJ, Chess PR, Curran ML, Dammann CEL, High P, Hsu DC, Pitts S, Sauer C, Srivastava S, Mink RB. Achieving Entrustable Professional Activities During Fellowship. Pediatrics 2021; 148:peds.2021-050196. [PMID: 34667096 DOI: 10.1542/peds.2021-050196] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/12/2021] [Indexed: 11/24/2022] Open
Abstract
BACKGROUND AND OBJECTIVES Entrustable Professional Activities (EPAs) were developed to assess pediatric fellows. We previously showed that fellowship program directors (FPDs) may graduate fellows who still require supervision. How this compares with their expectations for entrustment of practicing subspecialists is unknown. METHODS We surveyed US FPDs in 14 pediatric subspecialties through the Subspecialty Pediatrics Investigator Network between April and August 2017. For each of 7 common pediatric subspecialty EPAs, we compared the minimum level of supervision that FPDs required for graduation with the level they expected of subspecialists for safe and effective practice using the Friedman rank sum test and paired t test. We compared differences between subspecialties using linear regression. RESULTS We collected data from 660 FPDs (response rate 82%). For all EPAs, FPDs did not require fellows to reach the level of entrustment for graduation that they expected of subspecialists to practice (P < .001). FPDs expected the least amount of supervision for the EPAs consultation and handovers. Mean differences between supervision levels for graduation and practice were smaller for clinical EPAs (consultation, handovers, lead a team) when compared with nonclinical EPAs (quality improvement, management, lead the profession and scholarship; P = .001) and were similar across nearly all subspecialties. CONCLUSIONS Fellowship graduates may need continued development of clinical and nonclinical skills in their early practice period, underscoring a need for continued assessment and mentoring. Graduation readiness must be based on clear requirements, with alignment of FPD expectations and regulatory standards, to ensure quality care for patients.
Collapse
Affiliation(s)
- Pnina G Weiss
- Department of Pediatrics, School of Medicine, Yale University, New Haven, Connecticut
| | - Alan Schwartz
- Departments of Medical Education and Pediatrics, College of Medicine, University of Illinois at Chicago, Chicago, Illinois
| | | | - Bruce E Herman
- Department of Pediatrics, School of Medicine, University of Utah, Salt Lake City, Utah
| | | | - Tandy Aye
- Department of Pediatrics, School of Medicine, Stanford University, Stanford, California
| | - Jill J Fussell
- Department of Pediatrics, University of Arkansas for Medical Sciences, Little Rock, Arkansas
| | - Jennifer Kesselheim
- Department of Pediatrics, Dana-Farber/Boston Children's Cancer and Blood Disorders Center and Harvard Medical School, Harvard University, Boston, Massachusetts
| | - John D Mahan
- Department of Pediatrics, College of Medicine, The Ohio State University, Columbus, Ohio
| | - Kathleen A McGann
- Department of Pediatrics, School of Medicine, Duke University, Durham, North Carolina
| | - Angela Myers
- Department of Pediatrics, Children's Mercy Kansas City and School of Medicine, University of Missouri-Kansas City, Kansas City, Missouri
| | - Diane E J Stafford
- Department of Pediatrics, School of Medicine, Stanford University, Stanford, California
| | | | - Megan L Curran
- Department of Pediatrics, University of Colorado, Denver, Colorado
| | | | - Pamela High
- Department of Pediatrics, The Warren Alpert Medical School, Brown University, Providence, Rhode Island
| | - Deborah C Hsu
- Department of Pediatrics, Baylor College of Medicine and Texas Children's Hospital, Houston, Texas
| | - Sarah Pitts
- Department of Pediatrics, Harvard Medical School, Harvard University, Boston, Massachusetts
| | - Cary Sauer
- Department of Pediatrics, School of Medicine, Emory University and Children's Healthcare of Atlanta, Atlanta, Georgia
| | | | - Richard B Mink
- Department of Pediatrics, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California.,The Lundquist Institute for Biomedical Innovation, Harbor-University of California, Los Angeles Medical Center, Torrance, California
| |
Collapse
|
37
|
Carraccio C, Martini A, Van Melle E, Schumacher DJ. Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:1332-1336. [PMID: 33769339 DOI: 10.1097/acm.0000000000004075] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention. METHOD A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively. RESULTS After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments. CONCLUSIONS Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.
Collapse
Affiliation(s)
- Carol Carraccio
- C. Carraccio was vice president for competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina, at the time of this study
| | - Abigail Martini
- A. Martini is clinical research coordinator, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
| | - Elaine Van Melle
- E. Van Melle is senior education scientist, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada
| | - Daniel J Schumacher
- D.J. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio
| |
Collapse
|
38
|
Bonachea EM. Crossing the Entrustment Chasm: Do We Trust Graduating Fellows to Practice without Supervision? ATS Sch 2021; 2:289-292. [PMID: 34667977 PMCID: PMC8518651 DOI: 10.34197/ats-scholar.2021-0099ed] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Affiliation(s)
- Elizabeth M Bonachea
- Division of Neonatology, Nationwide Children's Hospital, Columbus, Ohio; and
- Department of Pediatrics, The Ohio State University, Columbus, Ohio
| |
Collapse
|
39
|
Weiss P, Schwartz A, Carraccio C, Herman BE, Mink RB. Minimum Supervision Levels Required by Program Directors for Pediatric Pulmonary Fellow Graduation. ATS Sch 2021; 2:360-369. [PMID: 34667986 PMCID: PMC8519336 DOI: 10.34197/ats-scholar.2021-0013oc] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Accepted: 05/14/2021] [Indexed: 11/30/2022] Open
Abstract
Background: Entrustable professional activities (EPAs) define the essential tasks expected of subspecialists in unsupervised practice. Although EPAs have been piloted in some programs, their use for summative assessment of pediatric pulmonology fellows for graduation has not been studied. Objective: To determine the minimum level of supervision that pediatric pulmonary program directors (PDs) require of their fellows for graduation and compare it with the minimum level of supervision they expect for a practicing subspecialist for the five pediatric pulmonology EPAs. Methods: Using a modified Delphi approach, we developed supervision scales for the five pediatric pulmonology EPAs and conducted a national survey of pediatric pulmonary PDs in the United States through the Subspecialty Pediatric Investigators Network between April 2017 and August 2017. Results: Forty-six pediatric pulmonary PDs completed the survey, representing a response rate of 85%. The majority did not require fellows to be trusted to practice without supervision for graduation for any of the five EPAs (level 5); the median minimum level of supervision they required was 4, equating to indirect supervision for complex cases. The minimum level for graduation, defined by consensus as the level of supervision for which no more than 20% of PDs would want the level to be lower to allow a fellow to graduate, was 3, which corresponded to requiring supervision for both simple and complex cases. There was a statistically significant difference between the minimum level of supervision deemed necessary by PDs for graduation and for practice as a subspecialist for each of the EPAs. Conclusion: Most pediatric pulmonary PDs reported that they would graduate fellows who may still require indirect supervision for the five pediatric pulmonology EPAs. The findings suggest a need for stakeholders to reevaluate the structure and outcomes of training programs and ensure support for pediatric pulmonologists in their early practice period.
Collapse
Affiliation(s)
- Pnina Weiss
- Department of Pediatrics, Yale School of Medicine, New Haven, Connecticut
| | - Alan Schwartz
- Department of Medical Education and Department of Pediatrics, University of Illinois at Chicago College of Medicine, Chicago, Illinois
| | | | | | - Richard B. Mink
- David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California; and
- Lundquist Institute for Biomedical Innovation at Harbor-University of California, Los Angeles, Medical Center, Torrance, California
| |
Collapse
|
40
|
Caretta‐Weyer HA, Schumacher DJ. Missing the mark: Alternative sources of variation in graduate milestone achievement. AEM EDUCATION AND TRAINING 2021; 5:e10618. [PMID: 34222752 PMCID: PMC8237087 DOI: 10.1002/aet2.10618] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/11/2021] [Accepted: 04/28/2021] [Indexed: 06/13/2023]
Affiliation(s)
- Holly A. Caretta‐Weyer
- Department of Emergency MedicineStanford University School of MedicineSan FranciscoCAUSA
| | - Daniel J. Schumacher
- Cincinnati Children’s Hospital Medical CenterUniversity of Cincinnati College of MedicineCincinnatiOHUSA
| |
Collapse
|
41
|
Sebok-Syer SS, Gingerich A, Holmboe ES, Lingard L, Turner DA, Schumacher DJ. Distant and Hidden Figures: Foregrounding Patients in the Development, Content, and Implementation of Entrustable Professional Activities. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S76-S80. [PMID: 34183606 DOI: 10.1097/acm.0000000000004094] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Entrustable professional activities (EPAs) describe activities that qualified professionals must be able to perform to deliver safe and effective care to patients. The entrustable aspect of EPAs can be used to assess learners through documentation of entrustment decisions, while the professional activity aspect can be used to map curricula. When used as an assessment framework, the entrustment decisions reflect supervisory judgments that combine trainees' relational autonomy and patient safety considerations. Thus, the design of EPAs incorporates the supervisor, trainee, and patient in a way that uniquely offers a link between educational outcomes and patient outcomes. However, achieving a patient-centered approach to education amidst both curricular and assessment obligations, educational and patient outcomes, and a supervisor-trainee-patient triad is not simple nor guaranteed. As medical educators continue to advance EPAs as part of their approach to competency-based medical education, the authors share a critical discussion of how patients are currently positioned in EPAs. In this article, the authors examine EPAs and discuss how their development, content, and implementation can result in emphasizing the trainee and/or supervisor while unintentionally distancing or hiding the patient. They consider creative possibilities for how EPAs might better integrate the patient as finding ways to better foreground the patient in EPAs holds promise for aligning educational outcomes and patient outcomes.
Collapse
Affiliation(s)
- Stefanie S Sebok-Syer
- S.S. Sebok-Syer is instructor, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: http://orcid.org/0000-0002-3572-5971
| | - Andrea Gingerich
- A. Gingerich is assistant professor, Division of Medical Sciences, University of Northern British Columbia, Prince George, British Columbia, Canada; ORCID: http://orcid.org/0000-0001-5765-3975
| | - Eric S Holmboe
- E.S. Holmboe is chief research, milestones development, and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: http://orcid.org/0000-0003-0108-6021
| | - Lorelei Lingard
- L. Lingard is professor, Department of Medicine and Faculty of Education, and senior scientist, Centre for Education, Research, and Innovation, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada; ORCID: http://orcid.org/0000-0002-1524-0723
| | - David A Turner
- D.A. Turner is vice president for competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina
| | - Daniel J Schumacher
- D.J. Schumacher is associate professor, Department of Pediatrics, Cincinnati Children's Hospital Medical Center and the University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: http://orcid.org/0000-0001-5507-8452
| |
Collapse
|
42
|
Van Melle E, Hall AK, Schumacher DJ, Kinnear B, Gruppen L, Thoma B, Caretta-Weyer H, Cooke LJ, Frank JR. Capturing outcomes of competency-based medical education: The call and the challenge. MEDICAL TEACHER 2021; 43:794-800. [PMID: 34121596 DOI: 10.1080/0142159x.2021.1925640] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?
Collapse
Affiliation(s)
- Elaine Van Melle
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Family Medicine, Queen's University, Kingston, Canada
| | - Andrew K Hall
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Emergency Medicine, Queen's University, Kingston,Canada
| | - Daniel J Schumacher
- Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Benjamin Kinnear
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Larry Gruppen
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI, USA
| | - Brent Thoma
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Canada
| | - Holly Caretta-Weyer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, USA
| | - Lara J Cooke
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Clinical Neurosciences, Division of Neurology, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Jason R Frank
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
43
|
Schumacher DJ, Caretta-Weyer H, Busari J, Carraccio C, Damodaran A, Gruppen LD, Hall AK, Kinnear B, Warm E, Ten Cate O. Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic. MEDICAL TEACHER 2021; 43:810-816. [PMID: 34038645 DOI: 10.1080/0142159x.2021.1925098] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | | | - Jamiu Busari
- Maastricht University and A Consultant pediatrician and Dean, Horacio Oduber Hospital, Aruba
| | | | - Arvin Damodaran
- Prince of Wales Clinical School, Faculty of Medicine, UNSW Sydney, Sydney, Australia
| | | | - Andrew K Hall
- Department of Emergency Medicine, Queen's University, Ontario, KN, Canada
| | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Eric Warm
- University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Olle Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
44
|
Turner DA, Schwartz A, Carraccio C, Herman B, Weiss P, Baffa JM, Chess P, Curran M, Dammann C, High P, Hsu D, Pitts S, Sauer C, Aye T, Fussell J, Kesselheim J, Mahan J, McGann K, Myers A, Mink R. Continued Supervision for the Common Pediatric Subspecialty Entrustable Professional Activities May Be Needed Following Fellowship Graduation. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S22-S28. [PMID: 34183598 DOI: 10.1097/acm.0000000000004091] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
PURPOSE Entrustable professional activities (EPAs) are one approach to competency-based medical education (CBME), and 7 EPAs have been developed that address content relevant for all pediatric subspecialties. However, it is not known what level of supervision fellowship program directors (FPDs) deem necessary for graduation. The Subspecialty Pediatrics Investigator Network (SPIN) investigated FPD perceptions of the minimum level of supervision required for a trainee to successfully graduate. METHOD In 2017, SPIN surveyed all FPDs of accredited fellowships for 14 subspecialties. For each EPA, the minimum supervision level for graduation (ranging from observation only to unsupervised practice) was set such that no more than 20% of FPDs would accept a lower level. RESULTS The survey response rate was 82% (660/802). The minimum supervision level for graduation varied across the 7 EPAs from 2 (direct) to 4 (indirect for complex cases), with significant differences between EPAs. The percentage of FPDs desiring a lower minimum supervision level ranged from 3% to 17%. Compared with the 4 nonclinical EPAs (quality improvement, management, lead within the profession, scholarship), the 3 clinical EPAs (consultation, handover, lead a team) had higher minimum supervision graduation levels (P < .001), with less likelihood that an FPD would graduate a learner below their minimum level (P < .001). CONCLUSIONS Consensus among FPDs across all pediatric subspecialties demonstrates the potential need for ongoing supervision for graduates in all 7 common pediatric subspecialty EPAs after fellowship. As CBME programs are implemented, processes and infrastructure to support new graduates are important considerations for leaders.
Collapse
Affiliation(s)
- David A Turner
- D.A. Turner is vice president of competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina, and consulting professor, Duke University Medical Center, Durham, North Carolina. At the time this work was completed, he was associate professor, Duke University Medical Center, Durham, North Carolina
| | - Alan Schwartz
- A. Schwartz is the Michael Reese Endowed Professor of Medical Education, interim head, Department of Medical Education, and research professor, Department of Pediatrics, University of Illinois at Chicago College of Medicine, Chicago, Illinois
| | - Carol Carraccio
- C. Carraccio was vice president of competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina at the time this work was completed
| | - Bruce Herman
- B. Herman is professor and vice chair of education, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, Utah
| | - Pnina Weiss
- P. Weiss is vice chair of education and associate professor, Department of Pediatrics, Yale School of Medicine, New Haven, Connecticut
| | - Jeanne M Baffa
- J.M. Baffa is emeritus associate professor of pediatrics, Sidney Kimmel Medical College of Thomas Jefferson University, Nemours/Alfred I. duPont Hospital for Children, Wilmington, Delaware
| | - Patricia Chess
- P. Chess is professor, University of Rochester Medical Center, Rochester, New York
| | - Megan Curran
- M. Curran is associate professor, Department of Pediatrics, University of Colorado-Denver, Denver, Colorado
| | - Christiane Dammann
- C. Damman is professor, Department of Pediatrics, Tufts Children's Hospital, Boston, Massachusetts
| | - Pamela High
- P. High is professor, the Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | - Deborah Hsu
- D. Hsu is professor, Department of Pediatrics, Section of Emergency Medicine, Baylor College of Medicine/Texas Children's Hospital, Houston, Texas
| | - Sarah Pitts
- S. Pitts is assistant professor, Department of Pediatrics, Harvard Medical School, Boston, Massachusetts
| | - Cary Sauer
- C. Sauer is associate professor, Emory University School of Medicine, Children's Healthcare of Atlanta, Atlanta, Georgia
| | - Tandy Aye
- T. Aye is associate professor, Department of Pediatrics, Stanford University School of Medicine, Stanford, California
| | - Jill Fussell
- J. Fussell is professor, Department of Pediatrics, University of Arkansas for Medical Sciences, Little Rock, Arkansas
| | - Jennifer Kesselheim
- J. Kesselheim is associate professor, Department of Pediatrics, Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School, Boston, Massachusetts
| | - John Mahan
- J. Mahan is professor, Department of Pediatrics, The Ohio State University College of Medicine, Columbus, Ohio
| | - Kathleen McGann
- K. McGann is vice chair of education and professor, Department of Pediatrics, Duke University Medical Center, Durham, North Carolina
| | - Angie Myers
- A. Myers is professor of pediatrics, University of Missouri-Kansas City School of Medicine, Kansas City, Missouri
| | - Richard Mink
- R. Mink is professor of pediatrics, The David Geffen School of Medicine at University of California Los Angeles (UCLA) and The Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, Torrance, California
| |
Collapse
|
45
|
Carraccio C. Harnessing the Potential Futures of CBME Here and Now. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S6-S8. [PMID: 34183595 DOI: 10.1097/acm.0000000000004102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The COVID-19 pandemic of 2020 exposed the reactive nature of the medical education community in response to a disruption that, at one time, may have seemed preposterous. In this article, the author reflected on the impact of an unpredictable plight on a system of medical education that (1) is continuous but doesn't function as a continuum and (2) requires adaptation but is steeped in a fixed mindset and structure that resists change. As a result, innovations which were previously considered impossible, such as time variable education and training, were forced into being. Inspired by the changes brought about by the pandemic, the ensuing decade is explored through a lens of possible futures to envision a path forward based on resilience rather than reactivity.
Collapse
Affiliation(s)
- Carol Carraccio
- C. Carraccio is a former pediatrician, clinician educator, program director, and researcher with a focus on medical education
| |
Collapse
|
46
|
Ten Cate O, Balmer DF, Caretta-Weyer H, Hatala R, Hennus MP, West DC. Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S96-S104. [PMID: 34183610 DOI: 10.1097/acm.0000000000004106] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.
Collapse
Affiliation(s)
- Olle Ten Cate
- O. ten Cate is professor of medical education and senior scientist, Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0002-6379-8780
| | - Dorene F Balmer
- D.F. Balmer is associate professor, Department of Pediatrics, The Children's Hospital of Philadelphia and University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0001-6805-4062
| | - Holly Caretta-Weyer
- H. Caretta-Weyer is assistant professor, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| | - Rose Hatala
- R. Hatala is professor, Department of Medicine, University of British Columbia, Vancouver, Canada; ORCID: https://orcid.org/0000-0003-0521-2590
| | - Marije P Hennus
- M.P. Hennus is a pediatric intensivist and program director, pediatric intensive care fellowship, University Medical Center Utrecht, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0003-1508-0456
| | - Daniel C West
- D.C. West is professor and senior director of medical education, Department of Pediatrics, Children's Hospital of Philadelphia and The Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0002-0909-4213
| |
Collapse
|
47
|
Schumacher DJ, Cate OT, Damodaran A, Richardson D, Hamstra SJ, Ross S, Hodgson J, Touchie C, Molgaard L, Gofton W, Carraccio C. Clarifying essential terminology in entrustment. MEDICAL TEACHER 2021; 43:737-744. [PMID: 33989100 DOI: 10.1080/0142159x.2021.1924365] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Olle Ten Cate
- Center for Research Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Arvin Damodaran
- Prince of Wales Clinical School, Faculty of Medicine, UNSW Sydney, Sydney, Australia
| | - Denyse Richardson
- Department of Medicine, University of Toronto, Toronto, Canada
- The Royal College of Physicians and Surgeons, Ottowa, Canada
| | - Stanley J Hamstra
- Accreditation Council for Graduate Medical Education, Chicago, IL, USA
- University of Ottawa, Ontario, Canada
- Department of Medical Education, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
| | - Shelley Ross
- Department of Family Medicine, University of Alberta, Edmonton, Canada
| | - Jennie Hodgson
- Virginia-Maryland College of Veterinary Medicine, Blacksburg, VA, USA
| | - Claire Touchie
- The Medical Council of Canada, University of Ottawa, Ottawa, Canada
| | - Laura Molgaard
- University of Minnesota College of Veterinary Medicine, St. Paul, MN, USA
| | - Wade Gofton
- Royal College of Physicians and Surgeons of Canada, University of Ottawa, Ottawa, Canada
| | | |
Collapse
|
48
|
Schwartz A, Borman-Shoap E, Carraccio C, Herman B, Hobday PM, Kaul P, Long M, O'Connor M, Mink R, Schumacher DJ, Turner DA, West DC. Learner Levels of Supervision Across the Continuum of Pediatrics Training. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S42-S49. [PMID: 34183601 DOI: 10.1097/acm.0000000000004095] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship. METHOD The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings. RESULTS Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency. CONCLUSIONS Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.
Collapse
Affiliation(s)
- Alan Schwartz
- A. Schwartz is Michael Reese Endowed Professor of Medical Education and research professor, pediatrics, University of Illinois College of Medicine, Chicago, Illinois, and network director, Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN), McLean, Virginia; ORCID: http://orcid.org/0000-0003-3809-6637
| | - Emily Borman-Shoap
- E. Borman-Shoap is associate professor, residency program director, and vice chair of education, Department of Pediatrics, University of Minnesota, Minneapolis, Minnesota; ORCID: https://orcid.org/0000-0001-7514-7793
| | - Carol Carraccio
- C. Carraccio was vice president for competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina, at the time the work was completed; ORCID: https://orcid.org/0000-0001-5473-8914
| | - Bruce Herman
- B. Herman is professor and vice chair of education, Department of Pediatrics, University of Utah, Salt Lake City, Utah
| | - Patricia M Hobday
- P.M. Hobday is assistant professor and director, Education in Pediatrics Across the Continuum (EPAC), Department of Pediatrics, University of Minnesota, Minneapolis, Minnesota
| | - Paritosh Kaul
- P. Kaul is professor and associate director, EPAC, Department of Pediatrics, University of Colorado, School of Medicine, Denver, Colorado; ORCID: https://orcid.org/0000-0003-4155-7406
| | - Michele Long
- M. Long is professor and director, EPAC, Department of Pediatrics, University of California, San Francisco, San Francisco, California; ORCID: https://orcid.org/0000-0002-8399-5589
| | - Meghan O'Connor
- M. O'Connor is assistant professor and director, EPAC, Department of Pediatrics, University of Utah, Salt Lake City, Utah; ORCID: https://orcid.org/0000-0002-5403-923X
| | - Richard Mink
- R. Mink is professor of pediatrics, David Geffen School of Medicine at the University of California, Los Angeles (UCLA), Los Angeles, California, pediatric critical care fellowship program director, Harbor-UCLA Medical Center, Torrance, California, and director, APPD Subspecialty Pediatrics Investigator Network, McLean, Virginia; ORCID: http://orcid.org/0000-0002-7998-4713
| | - Daniel J Schumacher
- D.J. Schumacher is associate professor, Department of Pediatrics, Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| | - David A Turner
- D.A. Turner was associate professor, Division of Pediatric Critical Care, Department of Pediatrics, and associate director, Graduate Medical Education, Duke University Hospital and Health System, Durham, North Carolina, at the time the work was completed
| | - Daniel C West
- D.C. West is professor and senior director of medical education, Department of Pediatrics, Children's Hospital of Philadelphia and The Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0002-0909-4213
| |
Collapse
|
49
|
Schumacher DJ, Turner DA. Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S1-S5. [PMID: 34183594 DOI: 10.1097/acm.0000000000004097] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- Daniel J Schumacher
- D.J. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - David A Turner
- D.A. Turner is vice president for competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina
| |
Collapse
|
50
|
Touchie C, Kinnear B, Schumacher D, Caretta-Weyer H, Hamstra SJ, Hart D, Gruppen L, Ross S, Warm E, Ten Cate O. On the validity of summative entrustment decisions. MEDICAL TEACHER 2021; 43:780-787. [PMID: 34020576 DOI: 10.1080/0142159x.2021.1925642] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.
Collapse
Affiliation(s)
- Claire Touchie
- Medical Council of Canada, Ottawa, Canada
- The University of Ottawa, Ottawa, Canada
| | - Benjamin Kinnear
- Internal Medicine and Pediatrics, University of Cincinnati College of Medicine/Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA
| | - Daniel Schumacher
- Pediatrics, Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Holly Caretta-Weyer
- Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, USA
| | - Stanley J Hamstra
- University of Toronto, Toronto, Ontario, Canada
- Accreditation Council for Graduate Medical Education, Chicago, IL, USA
| | - Danielle Hart
- Emergency Medicine, Hennepin Healthcare and the University of Minnesota, Minneapolis, MN, USA
| | - Larry Gruppen
- Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA
| | - Shelley Ross
- Department of Family Medicine, University of Alberta, Edmonton, AB, Canada
| | - Eric Warm
- University of Cincinnati College of Medicine Center, Cincinnati, OH, USA
| | - Olle Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|