Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities

Level of Supervision for the Entrustable Professional Activities Common to General Pediatrics and the Subspecialties Decreases from Residency to Fellowship

OBJECTIVES

To compare level of supervision (LOS) ratings of graduating pediatric residents with their assessments as fellows for the five Entrustable Professional Activities (EPAs) common to general pediatrics and the subspecialties and to determine if the difference between ratings from residency to fellowship is less for the QI and Practice Management EPAs, since the skills needed to perform these may be less context-dependent.

METHODS

We compared ratings of graduating residents with their assessments as fellows using LOS data from two sequential EPA studies.

RESULTS

There were 65 ratings from 41 residents at the first fellow assessment. At graduation, most residents needed little to no supervision for all EPAs with 94% (61/65) of ratings level four or five. In contrast, only 5/65 (8%) of the first fellow assessments were level four or five. The ratings difference for the QI and Practice Management EPAs was similar to the others.

CONCLUSIONS

LOS ratings for the EPAs common to generalists and subspecialists reset as residents become fellows. There was no evidence that the QI and Practice Management EPAs are less context-dependent. This study provides additional validity evidence for using these LOS scales to assess trainees in pediatric residency and fellowship.

Data sharing and big data in health professions education: Ottawa consensus statement and recommendations for scholarship

Changes in digital technology, increasing volume of data collection, and advances in methods have the potential to unleash the value of big data generated through the education of health professionals. Coupled with this potential are legitimate concerns about how data can be used or misused in ways that limit autonomy, equity, or harm stakeholders. This consensus statement is intended to address these issues by foregrounding the ethical imperatives for engaging with big data as well as the potential risks and challenges. Recognizing the wide and ever evolving scope of big data scholarship, we focus on foundational issues for framing and engaging in research. We ground our recommendations in the context of big data created through data sharing across and within the stages of the continuum of the education and training of health professionals. Ultimately, the goal of this statement is to support a culture of trust and quality for big data research to deliver on its promises for health professions education (HPE) and the health of society. Based on expert consensus and review of the literature, we report 19 recommendations in (1) framing scholarship and research through research, (2) considering unique ethical practices, (3) governance of data sharing collaborations that engage stakeholders, (4) data sharing processes best practices, (5) the importance of knowledge translation, and (6) advancing the quality of scholarship through multidisciplinary collaboration. The recommendations were modified and refined based on feedback from the 2022 Ottawa Conference attendees and subsequent public engagement. Adoption of these recommendations can help HPE scholars share data ethically and engage in high impact big data scholarship, which in turn can help the field meet the ultimate goal: high-quality education that leads to high-quality healthcare.

In EPAs we trust, is quality and safety a must? A cross-specialty analysis of entrustable professional activity guides

PURPOSE

The inclusion of quality improvement (QI) and patient safety (PS) into CanMEDS reflects an expectation that graduating physicians are competent in these areas upon training completion. To ensure that Canadian postgraduate specialty training achieves this, the translation of QI/PS competencies into training standards as part of the implementation of competency-based medical education requires special attention.

METHODS

We conducted a cross-specialty, multi-method analysis to examine how QI/PS was incorporated into the EPA Guides across 11 postgraduate specialties in Canada.

RESULTS

We identify cross-specialty variability in how QI/PS is incorporated, positioned, and emphasized in EPAs and milestones. QI/PS was primarily referenced alongside clinical activities rather than as a sole competency or discrete activity. Patterns were characterized in how QI/PS became incorporated into milestones through repetition and customization. QI/PS was also decoupled, conceptualized, and emphasized differently across specialties.

CONCLUSIONS

Variability in the inclusion of QI/PS in EPAs and milestones has important implications considering the visibility and influence of EPA Guides in practice. As specialties revisit and revise EPA Guides, there is a need to balance the standardization of foundational QI/PS concepts to foster shared understanding while simultaneously ensuring context-sensitive applications across specialties. Beyond QI/PS, this study illuminates the challenges and opportunities that lie in bridging theoretical frameworks with practical implementation in medical education, prompting broader consideration of how intrinsic roles and emergent areas are effectively incorporated into competency-based medical education.

Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment

ABSTRACT

In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.

Minimum Supervision Levels Required for Pediatric Cardiology Fellowship Graduation by Pediatric Cardiology Fellowship Program Directors

Entrustable professional activities (EPAs) are "observable essential tasks expected to be performed by a physician for safe patient care in practice." Six Pediatric Cardiology (PC) EPAs and their level of supervision (LOS) scales were developed by medical educators in PC using a modified Delphi process and reviewed by the Subspecialty Pediatrics Investigator Network (SPIN). However, their general use in assessment for PC fellows for graduation requirements has yet to be studied. The objective of this study was to determine the minimum LOS required for PC fellows to graduate and compare it with the minimum LOS expected for safe and effective practice for the six PC EPAs, from the perspective of the PC Fellowship Program Directors(FPD). All Fellowship Program Directors(FPD) of ACGME-accredited PC fellowships were surveyed through SPIN between April 2017 and August 2017. For each of the PC EPAs, the FPDs were asked to indicate the minimum LOS expected for graduation and whether they would allow a fellow to graduate if this level was not achieved and the minimum LOS expected for a practicing pediatric cardiologist to provide safe and effective patient care. The minimum LOS was defined as the LOS for which no more than 20% of FPDs would want a lower level. The survey response rate was 80% (47/59). The majority of the FPDs did not require a minimum LOS of five corresponding to unsupervised practice in any of the six PC EPAs at graduation. For EPAs related to imaging, arrhythmia management, and management of cardiac problems, the minimum LOS for graduation was 3, corresponding to being "trusted to perform a task with indirect supervision for most simple and a few complex cases." For the EPAs related to interventional cardiology, heart failure pulmonary hypertension, and cardiac intensive care, the minimum LOS for graduation was 2, corresponding to being "trusted to perform a task only with direct supervision and coaching." The minimum LOS considered necessary for safe and effective practice for all but one EPA was 3. For the EPA related to the management of cardiac problems, the minimum LOS for safe practice was 4, corresponding to being "trusted to execute tasks independently except for few complex and critical cases." Most PC FPDs reported they would not require fellows to achieve the highest entrustment level for any of the six PC EPAs for graduation. It is crucial that educational programs evolve to address these essential activities during training better and that stakeholders ensure that graduating PC fellows have adequate resources and infrastructure to continue professional development as early career pediatric cardiologists.

Faculty Decision Making in Ad Hoc Entrustment of Pediatric Critical Care Fellows: A National Case-Based Survey

Phenomenon: Ad hoc entrustment decisions reflect a clinical supervisor's estimation of the amount of supervision a trainee needs to successfully complete a task in the moment. These decisions have important consequences for patient safety, trainee learning, and preparation for independent practice. Determinants of these decisions have previously been described but have not been well described for acute care contexts such as critical care and emergency medicine. The ad hoc entrustment of trainees caring for vulnerable patient populations is a high-stakes decision that may differ from other contexts. Critically ill patients and children are vulnerable patient populations, making the ad hoc entrustment of a pediatric critical care medicine (PCCM) fellow a particularly high-stakes decision. This study sought to characterize how ad hoc entrustment decisions are made for PCCM fellows through faculty ratings of vignettes. The authors investigated how acuity, relationship, training level, and task interact to influence ad hoc entrustment decisions. Approach: A survey containing 16 vignettes that varied by four traits (acuity, relationship, training level, and task) was distributed to U.S. faculty of pediatric critical care fellowships in 2020. Respondents determined an entrustment level for each case and provided demographic data. Entrustment ratings were dichotomized by "high entrustment" versus "low entrustment" (direct supervision or observation only). The authors used logistic regression to evaluate the individual and interactive effects of the four traits on dichotomized entrustment ratings. Findings: One hundred seventy-eight respondents from 30 institutions completed the survey (44% institutional response rate). Acuity, relationship, and task all significantly influenced the entrustment level selected but did not interact. Faculty most frequently selected "direct supervision" as the entrustment level for vignettes, including for 24% of vignettes describing fellows in their final year of training. Faculty rated the majority of vignettes (61%) as "low entrustment." There was no relationship between faculty or institutional demographics and the entrustment level selected. Insights: As has been found in summative entrustment for pediatrics, internal medicine, and surgery trainees, PCCM fellows often rated at or below the "direct supervision" level of ad hoc entrustment. This may relate to declining opportunities to practice procedures, a culture of low trust propensity among the specialty, and/or variation in interpretation of entrustment scales.

Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment

BACKGROUND

Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment.

OBJECTIVES

We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME).

METHODS

In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale.

RESULTS

One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05).

CONCLUSIONS

We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.

Implementation of Entrustable Professional Activities in General Surgery: Results of a National Pilot Study

OBJECTIVE

The ongoing complexity of general surgery training has led to an increased focus on ensuring the competence of graduating residents. Entrustable professional activities (EPAs) are units of professional practice that provide an assessment framework to drive competency-based education. The American Board of Surgery convened a group from the American College of Surgeons, Accreditation Council for Graduate Medical Education (ACGME) Surgery Review Committee, and Association of Program Directors in Surgery to develop and implement EPAs in a pilot group of residency programs across the country. The objective of this pilot study was to determine the feasibility and utility of EPAs in general surgery resident training.

METHODS

5 EPAs were chosen based on the most common procedures reported in ACGME case logs and by practicing general surgeons (right lower quadrant pain, biliary disease, inguinal hernia), along with common activities covering additional ACGME milestones (performing a consult, care of a trauma patient). Levels of entrustment assigned (1 to 5) were observation only, direct supervision, indirect supervision, unsupervised, and teaching others. Participating in site recruitment and faculty development occurred from 2017 to 2018. EPA implementation at individual residency programs began on July 1, 2018, and was completed on June 30, 2020. Each site was assigned 2 EPAs to implement and collected EPA microassessments on residents for those EPAs. The site clinical competency committees (CCC) used these microassessments to make summative entrustment decisions. Data submitted to the independent deidentified data repository every 6 months included the number of microassessments collected per resident per EPA and CCC summative entrustment decisions.

RESULTS

Twenty-eight sites were selected to participate in the program and represented geographic and size variability, community, and university-based programs. Over the course of the 2-year pilot programs reported on 14 to 180 residents. Overall, 6,272 formative microassessments were collected (range, 0 to 1144 per site). Each resident had between 0 and 184 microassessments. The mean number of microassessments per resident was 5.6 (SD = 13.4) with a median of 1 [interquartile range (IQR) = 6]. There were 1,763 summative entrustment ratings assigned to 497 unique residents. The average number of observations for entrustment was 3.24 (SD 3.61) with a median of 2 (IQR 3). In general, PGY1 residents were entrusted at the level of direct supervision and PGY5 residents were entrusted at unsupervised practice or teaching others. For each EPA other than the consult EPA, the degree of entrustment reported by the CCC increased by resident level.

CONCLUSIONS

These data provide evidence that widespread implementation of EPAs across general surgery programs is possible, but variable. They provide meaningful data that graduating chief residents are entrusted by their faculty to perform without supervision for several common general surgical procedures and highlight areas to target for the successful widespread implementation of EPAs.

How Does TIMELESS Training Impact Resident Motivation for Learning, Assessment, and Feedback? Evaluating a Competency-Based Time-Variable Training Pilot

PURPOSE

As competency-based medical education has become the predominant graduate medical education training model, interest in time-variable training has grown. Despite multiple competency-based time-variable training (CBTVT) pilots ongoing in the United States, little is known about how this training approach impacts learners. The authors aim to explore how their CBTVT pilot program impacted resident motivation for learning, assessment, and feedback.

METHOD

The authors performed a qualitative educational case study on the Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) program at the University of Cincinnati from October 2020 through March 2022. Semistructured interviews were conducted with TIMELESS residents (n = 9) approximately every 6 months to capture experiences over time. The authors used inductive thematic analysis to develop themes and compared their findings with existing theories of learner motivation.

RESULTS

The authors developed 2 themes: TIMELESS had variable effects on residents' motivation for learning and TIMELESS increased resident engagement with and awareness of the program of assessment. Participants reported increased motivation to learn and seek assessment, though some felt a tension between performance (e.g., advancement through the residency program) and growth (e.g., improvement as a physician). Participants became more aware of the quality of assessments they received, in part due to TIMELESS increasing the perceived stakes of assessment, and reported being more deliberate when assessing other residents.

CONCLUSIONS

Resident motivation for learning, assessment, and feedback was impacted in ways that the authors contextualize using current theories of learner motivation (i.e., goal orientation theory and attribution theory). Future research should investigate how interventions, such as coaching, guided learner reflection, or various CBTVT implementation strategies, can help keep learners oriented toward mastery learning rather than toward performance.

The Autonomy Toolbox: A Multicenter Collaborative to Promote Resident Autonomy

OBJECTIVES

Autonomy is necessary for resident professional development and well-being. A recent focus on patient safety has increased supervision and decreased trainee autonomy. Few validated interventions exist to improve resident autonomy. We aimed to use quality improvement methods to increase our autonomy metric, the Resident Autonomy Score (RAS), by 25% within 1 year and sustain for 6 months.

METHODS

We developed a bundled-intervention approach to improve senior resident (SR) perception of autonomy on Pediatric Hospital Medicine (PHM) services at 5 academic children's hospitals. We surveyed SR and PHM faculty perceptions of autonomy and targeted interventions toward areas with the highest discordance. Interventions included SR and faculty development, expectation-setting huddles, and SR independent rounding. We developed a Resident Autonomy Score (RAS) index to track SR perceptions over time.

RESULTS

Forty-six percent of SRs and 59% of PHM faculty completed the needs assessment survey querying how often SRs were afforded opportunities to provide autonomous medical care. Faculty and SR ratings were discordant in these domains: SR input in medical decisions, SR autonomous decision-making in straightforward cases, follow-through on SR plans, faculty feedback, SR as team leader, and level of attending oversight. The RAS increased by 19% (3.67 to 4.36) 1 month after SR and faculty professional development and before expectation-setting and independent rounding. This increase was sustained throughout the 18-month study period.

CONCLUSIONS

SRs and faculty perceive discordant levels of SR autonomy. We created an adaptable autonomy toolbox that led to sustained improvement in perception of SR autonomy.

Clinical competency committee perceptions of entrustable professional activities and their value in assessing fellows: A qualitative study of pediatric subspecialty program directors

OBJECTIVES

To examine the composition and processes of Clinical Competency Committees (CCCs) assigning entrustable professional activity (EPA) levels of supervision for pediatric subspecialty fellows and to examine fellowship program director (FPD) perspectives about using EPAs to determine fellows' graduation readiness.

METHODS

A qualitative study was performed using one-on-one interviews with a purposeful sample of pediatric subspecialty FPDs to yield a thematic analysis. Semi-structured interview guides were used for participants who self-identified as EPA users or non-users. Inductive analysis and coding were performed on transcripts until theoretical sufficiency was attained.

RESULTS

Twenty-eight FPDs were interviewed. There was significant variability in the composition and processes of CCCs across subspecialties. FPDs felt that CCCs intuitively understand what entrustment means, allowing for ease of application of level of supervision (LOS) scales and consensus. FPDs perceived that EPAs provided a global assessment of fellows and are one tool to determine graduation readiness.

CONCLUSIONS

Although there was variability in the makeup and processes of CCCs across subspecialties, FPDs believe EPAs are intuitive and relatively easy to implement. Consensus can be reached easily using EPA-specific LOS scales focusing on entrustment. FPDs desire a better understanding of how EPAs should be used for graduation.

A Virtual Reality Curriculum to Enhance Residents' Behavioral Health Anticipatory Guidance Skills: A Pilot Trial

OBJECTIVE

Effective educational strategies to train and assess residents' skills in behavioral health competencies have been limited. We hypothesized that a training curriculum using virtual reality (VR) simulations would enhance residents' evidence-based skills related to behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI).

METHODS

We conducted a randomized controlled pilot trial of our curriculum within an urban academic clinic. Intervention and control groups received four 15-minute, monthly didactics on BHAG and MI followed by VR simulations to 1) deliberately practice delivering BHAG by verbally counseling avatars, who responded real-time (intervention group) or 2) assessing pediatric respiratory distress (control group). Two months post-VR training, all residents participated in a unique VR behavioral health scenario which was recorded and coded by pediatric psychologists via an observation instrument to assess residents' skills. Differences in BHAG and MI competencies were assessed via independent samples t tests for continuous variables and Fisher's exact tests for categorical variables.

RESULTS

Sixty-six percent (n = 55) of eligible residents completed all study protocols comprising 35 intervention and 20 control participants. Intervention participants demonstrated an increase in BHAG skills (P = .002), percentage of open-ended questions asked (P = .04), and percentage of MI adherent behaviors (P = .04) when compared to the control group.

CONCLUSIONS

VR may be an effective educational strategy for residents to acquire BHAG and MI skills though deliberate practice. Next steps will focus on assessing the generalizability of the VR curriculum as well as exploring the opportunities to enhance its scalability through artificial intelligence.

Developing an Entrustable Professional Activity to Improve the Care of LGBTQ+ Youth

Despite the multitude of health inequities often experienced by lesbian, gay, bisexual, transgender, and queer (LGBTQ)+ youth, few training programs consistently and comprehensively address these disparities through education and assessment of learners. Over the past decade, programs have begun to incrementally increase the number of hours dedicated to LGBTQ+ health training; however, the lack of standardization for curricula and assessment remains a significant barrier. As medical education shifts to a competency-based approach, focusing on patient outcomes and the competencies trainees must achieve to meet those outcomes, we must take this opportunity to develop curricula and assessment tools that emphasize inclusive, affirming, equitable, and culturally humble care for LGBTQ+ youth. Entrustable professional activities (EPA) are increasingly used to educate and assess trainees on their progression from full supervision to unsupervised practice. Given their multifaceted use to standardize curricula, assess learners, and highlight programmatic gaps in education, we believe an EPA is the ideal tool to systematically improve LGBTQ+ health training in graduate medical education. Our team of national experts in this field developed an LGBTQ+ health EPA. We describe our methods of development, the functions and curricular components of the EPA, and the many applications for widespread program use.

Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020

IMPORTANCE

Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency.

OBJECTIVE

To assess progress in developing an entrustment process in the Core EPAs framework.

DESIGN, SETTING, AND PARTICIPANTS

In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database.

INTERVENTIONS

Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations.

MAIN OUTCOMES AND MEASURES

On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded.

RESULTS

Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs.

CONCLUSIONS AND RELEVANCE

These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.

Making prospective entrustment decisions: Knowing limits, seeking help and defaulting

INTRODUCTION

Many studies focused on entrustment have not considered prospective entrustment decisions, where clinical competency and entrustment committees determine how much supervision trainees will require in the future for tasks occurring in not completely known contexts. The authors sought to explore factors that influence and determine prospective entrustment decisions made by members of such committees in graduate medical education (GME) and undergraduate medical education (UME).

METHODS

The authors conducted a constructivist grounded theory study with 23 faculty participants from GME and UME clinical competency and entrustment committees in the United States between October 2020 and March 2021. Interviews sought to explore factors and considerations participants weigh in making prospective entrustment decisions about trainees. Data collection and analysis occurred in an iterative fashion, ensuring constant comparison. Theoretical sampling was used to confirm, disconfirm and elaborate on the evolving results.

RESULTS

Trainees' ability to know limits and seek help is the foundation of participants' prospective entrustment decision making. Most participants, however, describe a presumption of trainee readiness to progress and describe commonly making default prospective entrustment decisions unless 'red flags' in performance are present. Although participants desire sufficient and trusted data about trainee performance to inform decisions, they often lack it. Finally, the perceived permanence and consequences of prospective entrustment decisions influence how participants weigh other factors contributing to prospective entrustment decisions.

CONCLUSION

Trainees' ability to know limits and seek help appears to be the foundation of prospective entrustment decision making. Training programmes should strive to collect and employ robust data supporting and questioning the presence of these attributes.

Pediatric Emergency Medicine Fellowship Program Directors' Viewpoint: Minimum Levels of Entrustment for Graduating Fellows and Practicing Physicians to Perform the Subspecialty's Professional Activities

OBJECTIVES

Entrustable Professional Activities (EPAs) are essential tasks physicians perform within their professions. Entrustment levels that pediatric emergency medicine (PEM) fellowship program directors (FPDs) expect graduating fellows to achieve for PEM-specific and common pediatric subspecialty EPAs remain unreported. This study aims to determine minimum entrustment levels FPDs require fellows to achieve to graduate from fellowship and to compare FPD expectations for fellows versus practicing PEM physicians.

METHODS

Secondary analysis of PEM-specific data from a national multispecialty cross-sectional survey of pediatric subspecialty FPDs. For 6 PEM-specific and 7 common pediatric subspecialty EPAs, PEM FPDs indicated (1) minimum entrustment levels fellows should achieve by training completion, (2) whether they would allow a fellow to graduate below these minimum levels, and (3) minimum levels for safe and effective practice by PEM physicians. Minimum levels were defined as the level that more than 80% of FPDs would not drop below.

RESULTS

Sixty of 77 PEM FPDs (78%) completed the survey. Most respondents did not require fellows to achieve the highest level (level 5-no supervision) by graduation for any PEM-specific EPAs. The median level FPDs expected for practicing PEM physicians was 5 (trusted to perform without supervision) for EPAs 1 and 4 and level 4 (indirect supervision for complex cases) for the remaining PEM-specific EPAs. Minimum levels expected by FPDs for common subspecialty pediatric EPAs were lower for both groups.

CONCLUSIONS

Most PEM FPDs indicated that they would graduate fellows before their achievement of the highest entrustment level for all EPAs. Most also indicated that they do not expect practicing PEM physicians to perform all EPAs without supervision. These findings indicate need for stakeholders to evaluate current structure and outcomes of PEM fellowship programs and for institutions and organizations to ensure adequate support in time and resources for ongoing learning for practicing PEM physicians.

Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activities Across Subspecialties

OBJECTIVE

To determine the relationship between level of supervision (LOS) ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) with their associated subcompetency milestones across subspecialties and by fellowship training year.

METHODS

Clinical Competency Committees (CCCs) in 14 pediatric subspecialties submitted LOS ratings for 6 Common Subspecialty EPAs and subcompetency milestone levels mapped to these EPAs. We examined associations between these subcompetency milestone levels and LOS ratings across subspecialty training year by fitting per-EPA linear mixed effects models, regressing LOS rating on milestone level and on training year.

RESULTS

CCCs from 211 pediatric fellowship programs provided data for 369 first, 336 second, and 331 third year fellows. Mean subcompetency milestone levels increased similarly among subspecialties for most EPAs compared with the reference, Adolescent Medicine. Mean subcompetency milestones mapped to each EPA and mean EPA LOS ratings generally increased by training year across all subspecialties.

CONCLUSIONS

Subcompetency milestones levels mapped to each Common Subspecialty EPA and the EPA LOS ratings increase similarly across subspecialties and by training year, providing validity evidence for using EPA LOS to assess pediatric subspecialty trainee performance. This study supports the development of tools to facilitated the CCC evaluation process across all pediatric subspecialties.

Perspectives on Complex Care Training in a Large Academic Pediatric Training Program

OBJECTIVE

To identify gaps and opportunities in complex care training for pediatric residents.

METHODS

Residents in an academic pediatric residency program were surveyed about: training experiences in complex care; self-entrustment in key clinical activities in complex care; educational strategies that would increase preparedness; and recommendations for curriculum development. We used descriptive statistics for quantitative data and content analysis for free-text responses.

RESULTS

Of the 160 residents surveyed, 110 (69%) participated. Most participants reported prior clinical exposure to children with medical complexity (CMC; 106, 96%) during both inpatient (82, 75%) and outpatient (88, 80%) clinical rotations. Mean self-entrustment was at or below "somewhat confident" for all clinical activities in complex care, for residents in all postgraduate years. Clinical activities with highest reported self-entrustment included evaluating aspiration into the airway, nutritional issues, care coordination, and evaluating pain. Lowest self-entrustment was reported for facilitating transition to adult care, managing medical technologies, and safety/emergency planning. In terms of educational strategies, participants recommended inpatient encounters with an expert preceptor teaching about evaluating aspiration, pain/irritability and dysmotility (>50%); discussions with patients/families for advocacy, difficult discussions, and transition to adult care (>40%); and hands-on practice for medical technology care (>40%).

CONCLUSIONS

Pediatric residents report limited self-entrustment in performing key clinical activities in complex care, including for residents at the end of their last postgraduate year. Future curriculum development should prioritize direct observation of clinical encounters with CMC by expert preceptors, partnership with patients and families of CMC, and hands-on simulation.

Physical Therapist Residency Competency-Based Education: Development of an Assessment Instrument

OBJECTIVE

The objective of this study was to develop generic domains of competence (DoC) with associated competencies and milestones for physical therapist residency education. This work was intended to culminate in establishing validity evidence to support a competency-based assessment instrument that could be used by residency programs to evaluate resident achievement of the competencies, regardless of specialty area.

METHODS

Employing the modified Delphi method, a residency education work group developed an evaluation instrument that included 7 DoC and 31 associated competencies with 6 milestones as rating scales for each competency. The instrument was distributed to mentors and residents in accredited physical therapist residency programs to establish validity evidence. Evaluations (measured by milestones) and demographics were collected at 3 time points (program entry, midterm, and final). Scores across these time points were compared using Kruskal-Wallis tests. Reliability was assessed with kappa statistics (interrater reliability) and alpha reliability coefficients (internal consistency). Construct validity was examined using confirmatory factor analysis via structural equation modeling.

RESULTS

Overall, 237 mentors and 228 residents completed 824 evaluations (460 by mentors and 364 resident self-evaluations) across the time points. Scores significantly increased from entry through final time points. The interrater reliability of the associated behaviors ranged from moderate to substantial agreement (κ = 0.417-0.774). The internal consistency was high for all DoC at every time point (α reliability coefficients = .881-.955 for entry, .857-.925 for midterm, and .824-.902 for final). After confirmatory factor analysis with structural equation modeling was performed, a model that included 7 DoC and 20 associated competencies was proposed.

CONCLUSIONS

The residency assessment instrument developed demonstrates interrater reliability and validity evidence and therefore supports competency-based assessment of resident clinical performance across specialty areas. Additionally, the instrument aligns the physical therapy profession with other professions, such as medicine, dentistry, and pharmacy, that have transitioned to competency-based education.

IMPACT

This study outlines the benefits of moving to competency-based education for physical therapist residents, using a sound evaluation tool that evaluates residents across specialty areas. The instrument will allow for transition to competency-based education in physical therapist residency education programs.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters

PURPOSE

In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales.

METHOD

The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments.

RESULTS

Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs.

CONCLUSIONS

In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

A Multistakeholder Approach to the Development of Entrustable Professional Activities in Complex Care

BACKGROUND

Care of children with medical complexity (CMC) involves unique knowledge, skills, and attitudes. Previous work has identified curricular priorities in complex care for pediatricians yet there are no established competency frameworks to guide curriculum development. We aimed to develop and appraise Entrustable Professional Activities (EPAs) for the care of CMC with multistakeholder involvement.

METHODS

We recruited complex care practitioners to develop EPAs using a template for elaborating descriptive elements. A team of clinicians, educators, trainees, and family leaders refined EPAs and mapped content to the Accreditation Council for Graduate Medical Education Milestones. We conducted virtual focus groups to assess whether EPAs represented the essential skills of pediatricians caring for CMC. Focus group data were analyzed using content analysis.

RESULTS

Content experts developed 11 EPAs for the care of CMC describing knowledge, skills, and attitudes required for attaining competency. EPAs were mapped to 21 of the 21 (100%) reporting pediatric milestones. Focus group participants endorsed and refined EPA content. Categories of feedback included clarifying medical knowledge, expanding on interpersonal communication skills, emphasizing systems-based practice, and affirming family partnership.

CONCLUSIONS

A systematic approach to developing EPAs for the care of CMC provides a guide for curriculum development and assessment in complex care.

Psychiatric Education for Pediatric Emergency Medicine Fellows: Should We be Doing More?

The number of psychiatric encounters is steadily increasing across all pediatric emergency departments. Based on this growing national crisis, we hypothesized there is a lack of robust education in behavioral/mental health emergencies during pediatric emergency medicine (PEM) fellowship.

METHODS

This was a cross-sectional survey of PEM fellowship directors. The survey was submitted to the Pediatric Emergency Medicine Program Director Survey Committee via REDCap to all 83 fellowship directors.

RESULTS

Fifty-five (66%) PEM fellowship directors completed the survey from September 2019 to December 2019. Forty-three (78%) programs provide some type of didactic teaching, although only 7 (13%) programs offer more than 5 hours of didactics over 3 years of fellowship. Six (11%) programs offer electronic resources, 13 (24%) offer simulations in psychiatry, and only 4 (7%) programs require any type of psychiatry rotation. Forty-six (84%) programs do not currently use psychiatric emergencies as a competency for evaluating PEM fellows. Program directors felt significantly more confident in faculty than graduating fellows in the management of numerous diagnoses including agitation (P < 0.001), homicidal ideation (P = 0.044), neuroleptic malignant syndrome (P = 0.006), acute dystonia (P < 0.001), and heroin overdose (P = 0.022). Ninety-six percent felt online curricula would be helpful to address this knowledge gap.

CONCLUSIONS

Despite growing behavioral/mental health needs in the pediatric emergency department, there continues to be opportunity to improve psychiatric education during PEM fellowship. Program directors felt more confident in faculty than graduating fellows with several psychiatric diagnoses. Pediatric emergency medicine fellowship directors expressed a strong interest in online tools to assist in the education of PEM fellows.

Entrustable Professional Activities for Chinese Standardized Residency Training in Pediatric Intensive Care Medicine

BACKGROUND

Entrustable professional activities (EPAs) were first introduced by Olle ten Cate in 2005. Since then, hundreds of applications in medical research have been reported worldwide. However, few studies discuss the use of EPAs for residency training in pediatric intensive care medicine. We conducted a pilot study of EPA for pediatric intensive care medicine to evaluate the use of EPAs in this subspecialty.

MATERIALS AND METHODS

A cross-sectional study was implemented in pediatric intensive care medicine standardized residency training at the Qilu Hospital of Shandong University. An electronic survey assessing EPA performance using eight scales composed of 15 categories were distributed among residents and directors.

RESULTS

A total of 217 director-assessment and 44 residents' self-assessment questionnaires were collected, both demonstrating a rising trend in scores across postgraduate years. There were significant differences in PGY1-vs.-PGY2 and PGY1-vs.-PGY3 director-assessment scores, while there were no differences in PGY2-vs.-PGY3 scores. PGY had a significant effect on the score of each EPA, while position significantly affected the scores of all EPAs except for EPA1 (Admit a patient) and EPA2 (Select and interpret auxiliary examinations). Gender only significantly affected the scores of EPA6 (Report a case), EPA12 (Perform health education), and EPA13 (Inform bad news).

CONCLUSION

This study indicates that EPA assessments have a certain discriminating capability among different PGYs in Chinese standardized residency training in pediatric intensive care medicine. Postgraduate year, gender, and resident position affected EPA scores to a certain extent. Given the inconsistency between resident-assessed and director-assessed scores, an improved feedback program is needed in the future.

The Use of Entrustable Professional Activities in Pediatric Postgraduate Medical Education: A Systematic Review

BACKGROUND

Entrustable Professional Activities (EPAs) provide a framework to make judgments of trainees' abilities in several settings including postgraduate medical education. No systematic review of the role of EPAs in pediatrics has yet been performed.

OBJECTIVES

In our systematic review, we sought to determine the use of EPAs in pediatrics to identify research gaps, summarize and discuss evidence relating to the development, implementation, and assessment.

DATA SOURCES

Medline, Scopus, PsycINFO, MedEdPortal, and Web of Science.

STUDY ELIGIBILITY CRITERIA

Two independent reviewers used a structured screening protocol in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and Association for Medical Education in Europe's guide for systematic reviews to include all articles reporting on EPAs in postgraduate medical education and pediatrics in particular. There were no restrictions due to language, study design, or participants.

STUDY APPRAISAL AND SYNTHESIS METHODS

Data on development, implementation, feasibility, acceptance, and assessment of EPAs were extracted and analyzed by the 2 independent researchers.

RESULTS

Twenty-eight articles published between 2014 and 2020 were included in the review. We found an increase in publications and a notable shift from descriptions of development processes toward aspects beyond development, ie, implementation, feasibility, acceptance/perception, and assessment.

LIMITATIONS

Studies from non-English-speaking countries are scarce which might lead to an inaccurate representation of actual international practice.

CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS

We provide a comprehensive overview of EPAs in pediatrics to guide future curriculum developers in collaborative development, implementation and assessment of EPAs in pediatric postgraduate medical education.

Assessment and Feedback Methods in Competency-Based Medical Education

Graduate and continuing medical education programs have transitioned to a competency-based medical education (CBME) model, where abilities necessary for the practice of medicine, called competencies, are defined as learning outcomes. Each competency contains milestones that describe stages of development from novice to master and provide criteria for assessing learners along this continuum. Assessments in CBME are formative and are intended to assist learners in achieving the next stage in mastery rather than render a final evaluation. Although written examinations are commonly used to evaluate learners, assessments measuring the performance of skills, such as direct observation or patient registries, provide stronger evidence of achievement of learning outcomes. Once assessments identify strengths and areas of improvement in a learner, effective feedback is critical in assisting their professional development. Feedback should be routine, timely, non-threatening, specific, and encourage self-assessment. The "Ask-Tell-Ask" feedback method follows this framework by asking for the learner's self-assessment first, telling the observer's own assessment next, and then finally asking for the learner's questions and action plan to address the issues raised. Effective assessment and feedback in competency based medical education assists learners in their continued professional development and aims to provide evidence of their impact on the health status of patients and communities.

An update on current EPAs in graduate medical education: A scoping review

The purpose of this scoping review is to update the recent progress of EPAs research in GME, focusing on the topical concern of EPAs effectiveness, and to provide a reference for medical researchers in countries/regions interested in introducing EPAs. Guided by Arksey and O'Malley's framework regarding scoping reviews, the researchers, in January 2021, conducted a search in five databases to ensure the comprehensiveness of the literature. After the predetermined process, 29 articles in total were included in this study. The most common areas for the implementation and evaluation of EPAs were Surgery (n = 7,24.1%), Pediatric (n = 5,17.2%) and Internal medicine (n = 4,13.8%), a result that shows a relatively large change in the research trend of EPAs in the last two years. Prior to 2018, EPAs research focused on internal medicine, psychiatry, family medicine, and primary care. The articles in the category of EPAs implementation and evaluation had four main themes: (1) validation of EPAs (n = 16,55.2%); (2) describing the experience of implementing EPAs (n = 11,37.9%); (3) examining the factors and barriers that influence the implementation and evaluation of EPAs (n = 6,20.6%); and (4) researching the experiences of faculty, interns, and other relevant personnel in using EPAs. Training programs were the most common EPAs implementation setting (n = 26,89.6%); direct observation and evaluation (n = 12,41.4%), and evaluation by scoring reports (n = 5,17.2%) were the two most common means of assessing physicians' EPA levels; 19 papers (65.5%) used faculty evaluation, and nine of these papers also used self-assessment (31.0%); the most frequently used tools in the evaluation of EPAs were mainly researcher-made instruments (n = 37.9%), assessment form (n = 7,24.1%), and mobile application (n = 6,20.7%). Although EPAs occupy an increasingly important place in international medical education, this study concludes that the implementation and diffusion of EPAs on a larger scale is still difficult.

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

IMPORTANCE

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.

OBJECTIVE

To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.

DESIGN, SETTING, AND PARTICIPANTS

This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.

INTERVENTIONS

Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.

MAIN OUTCOMES AND MEASURES

To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.

RESULTS

This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.

CONCLUSIONS AND RELEVANCE

These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

A tale of two frameworks: charting a path to lifelong learning for physician leaders through CanMEDS and LEADS

PURPOSE

The purpose of this paper was to determine the complementarity between the Canadian Medical Education Directions for Specialists (CanMEDS) physician competency and LEADS leadership capability frameworks from three perspectives: epistemological, philosophical and pragmatic. Based on those findings, the authors propose how the frameworks collectively layout pathways of lifelong learning for physician leadership.

DESIGN/METHODOLOGY/APPROACH

Using a qualitative approach combining critical discourse analysis with a modified Delphi, the authors examined "How complementary the CanMEDS and LEADS frameworks are in guiding physician leadership development and practice" with the following sub-questions: What are the similarities and differences between CanMEDS and LEADS from: An epistemological and philosophical perspective? The perspective of guiding physician leadership training and practice? How can CanMEDS and LEADS guide physician leadership development from medical school to retirement?

FINDINGS

Similarities and differences exist between the two frameworks from philosophical and epistemological perspectives with significant complementarity. Both frameworks are founded on a caring ethos and value physician leadership - CanMEDS (for physicians) and LEADS (physicians as one of many professions) define leadership similarly. The frameworks share beliefs in the function of leadership, embrace a belief in distributed leadership, and although having some philosophical differences, have a shared purpose (preparing for changing health systems). Practically, the frameworks are mutually supportive, addressing leadership action in different contexts and where there is overlap, complement one another in intent and purpose.

ORIGINALITY/VALUE

To the best of the authors' knowledge, this is the first paper to map the CanMEDS (physician competency) and LEADS (leadership capabilities) frameworks. By determining the complementarity between the two, synergies can be used to influence physician leadership capacity needed for today and the future.

Achieving Entrustable Professional Activities During Fellowship

BACKGROUND AND OBJECTIVES

Entrustable Professional Activities (EPAs) were developed to assess pediatric fellows. We previously showed that fellowship program directors (FPDs) may graduate fellows who still require supervision. How this compares with their expectations for entrustment of practicing subspecialists is unknown.

METHODS

We surveyed US FPDs in 14 pediatric subspecialties through the Subspecialty Pediatrics Investigator Network between April and August 2017. For each of 7 common pediatric subspecialty EPAs, we compared the minimum level of supervision that FPDs required for graduation with the level they expected of subspecialists for safe and effective practice using the Friedman rank sum test and paired t test. We compared differences between subspecialties using linear regression.

RESULTS

We collected data from 660 FPDs (response rate 82%). For all EPAs, FPDs did not require fellows to reach the level of entrustment for graduation that they expected of subspecialists to practice (P < .001). FPDs expected the least amount of supervision for the EPAs consultation and handovers. Mean differences between supervision levels for graduation and practice were smaller for clinical EPAs (consultation, handovers, lead a team) when compared with nonclinical EPAs (quality improvement, management, lead the profession and scholarship; P = .001) and were similar across nearly all subspecialties.

CONCLUSIONS

Fellowship graduates may need continued development of clinical and nonclinical skills in their early practice period, underscoring a need for continued assessment and mentoring. Graduation readiness must be based on clear requirements, with alignment of FPD expectations and regulatory standards, to ensure quality care for patients.

Minimum Supervision Levels Required by Program Directors for Pediatric Pulmonary Fellow Graduation

Background: Entrustable professional activities (EPAs) define the essential tasks expected of subspecialists in unsupervised practice. Although EPAs have been piloted in some programs, their use for summative assessment of pediatric pulmonology fellows for graduation has not been studied.

Objective: To determine the minimum level of supervision that pediatric pulmonary program directors (PDs) require of their fellows for graduation and compare it with the minimum level of supervision they expect for a practicing subspecialist for the five pediatric pulmonology EPAs.

Methods: Using a modified Delphi approach, we developed supervision scales for the five pediatric pulmonology EPAs and conducted a national survey of pediatric pulmonary PDs in the United States through the Subspecialty Pediatric Investigators Network between April 2017 and August 2017.

Results: Forty-six pediatric pulmonary PDs completed the survey, representing a response rate of 85%. The majority did not require fellows to be trusted to practice without supervision for graduation for any of the five EPAs (level 5); the median minimum level of supervision they required was 4, equating to indirect supervision for complex cases. The minimum level for graduation, defined by consensus as the level of supervision for which no more than 20% of PDs would want the level to be lower to allow a fellow to graduate, was 3, which corresponded to requiring supervision for both simple and complex cases. There was a statistically significant difference between the minimum level of supervision deemed necessary by PDs for graduation and for practice as a subspecialist for each of the EPAs.

Conclusion: Most pediatric pulmonary PDs reported that they would graduate fellows who may still require indirect supervision for the five pediatric pulmonology EPAs. The findings suggest a need for stakeholders to reevaluate the structure and outcomes of training programs and ensure support for pediatric pulmonologists in their early practice period.

Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended

PURPOSE

Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention.

METHOD

A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively.

RESULTS

After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments.

CONCLUSIONS

Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.

Distant and Hidden Figures: Foregrounding Patients in the Development, Content, and Implementation of Entrustable Professional Activities

Entrustable professional activities (EPAs) describe activities that qualified professionals must be able to perform to deliver safe and effective care to patients. The entrustable aspect of EPAs can be used to assess learners through documentation of entrustment decisions, while the professional activity aspect can be used to map curricula. When used as an assessment framework, the entrustment decisions reflect supervisory judgments that combine trainees' relational autonomy and patient safety considerations. Thus, the design of EPAs incorporates the supervisor, trainee, and patient in a way that uniquely offers a link between educational outcomes and patient outcomes. However, achieving a patient-centered approach to education amidst both curricular and assessment obligations, educational and patient outcomes, and a supervisor-trainee-patient triad is not simple nor guaranteed. As medical educators continue to advance EPAs as part of their approach to competency-based medical education, the authors share a critical discussion of how patients are currently positioned in EPAs. In this article, the authors examine EPAs and discuss how their development, content, and implementation can result in emphasizing the trainee and/or supervisor while unintentionally distancing or hiding the patient. They consider creative possibilities for how EPAs might better integrate the patient as finding ways to better foreground the patient in EPAs holds promise for aligning educational outcomes and patient outcomes.

Capturing outcomes of competency-based medical education: The call and the challenge

There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic

Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

Continued Supervision for the Common Pediatric Subspecialty Entrustable Professional Activities May Be Needed Following Fellowship Graduation

PURPOSE

Entrustable professional activities (EPAs) are one approach to competency-based medical education (CBME), and 7 EPAs have been developed that address content relevant for all pediatric subspecialties. However, it is not known what level of supervision fellowship program directors (FPDs) deem necessary for graduation. The Subspecialty Pediatrics Investigator Network (SPIN) investigated FPD perceptions of the minimum level of supervision required for a trainee to successfully graduate.

METHOD

In 2017, SPIN surveyed all FPDs of accredited fellowships for 14 subspecialties. For each EPA, the minimum supervision level for graduation (ranging from observation only to unsupervised practice) was set such that no more than 20% of FPDs would accept a lower level.

RESULTS

The survey response rate was 82% (660/802). The minimum supervision level for graduation varied across the 7 EPAs from 2 (direct) to 4 (indirect for complex cases), with significant differences between EPAs. The percentage of FPDs desiring a lower minimum supervision level ranged from 3% to 17%. Compared with the 4 nonclinical EPAs (quality improvement, management, lead within the profession, scholarship), the 3 clinical EPAs (consultation, handover, lead a team) had higher minimum supervision graduation levels (P < .001), with less likelihood that an FPD would graduate a learner below their minimum level (P < .001).

CONCLUSIONS

Consensus among FPDs across all pediatric subspecialties demonstrates the potential need for ongoing supervision for graduates in all 7 common pediatric subspecialty EPAs after fellowship. As CBME programs are implemented, processes and infrastructure to support new graduates are important considerations for leaders.

Harnessing the Potential Futures of CBME Here and Now

The COVID-19 pandemic of 2020 exposed the reactive nature of the medical education community in response to a disruption that, at one time, may have seemed preposterous. In this article, the author reflected on the impact of an unpredictable plight on a system of medical education that (1) is continuous but doesn't function as a continuum and (2) requires adaptation but is steeped in a fixed mindset and structure that resists change. As a result, innovations which were previously considered impossible, such as time variable education and training, were forced into being. Inspired by the changes brought about by the pandemic, the ensuing decade is explored through a lens of possible futures to envision a path forward based on resilience rather than reactivity.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

Clarifying essential terminology in entrustment

With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

Learner Levels of Supervision Across the Continuum of Pediatrics Training

PURPOSE

To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship.

METHOD

The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings.

RESULTS

Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency.

CONCLUSIONS

Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.

On the validity of summative entrustment decisions

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

A Novel Operative Coaching Program for General Surgery Chief Residents Improves Operative Efficiency

INTRODUCTION

We evaluated the effect of an operative coaching (OC) model on general surgery chief residents' operative efficiency (OE) measured by operative times. We hypothesized that higher levels of entrustment surgeons intend to offer resident in future similar cases are associated with improved OE.

MATERIALS AND METHODS

From July 2018 to June 2019, we used a validated instrument to score prospective resident entrustment in 228 evaluations of 6 chief residents during 12 OC sessions each (3 lap colectomy, 3 lap cholecystectomy, 3 ventral hernia, 3 inguinal hernia). Operative times of matched case CPT codes performed by coached chiefs (N = 500) were matched via CPT code to the cases of uncoached chiefs in the academic year 2016-2017 (N = 478). Statistical analysis was performed using Pearson correlation and one-way ANOVA.

RESULTS

Prospective entrustment scores from coached chief residents were associated with significantly shorter operative times in matched complex cases (CC) (r = -0.58, p = 0.0047). A similar trend was observed in noncomplex cases (NCC) (r = -0.29, p = 0.18). Compared to the historical cohort, coached chief residents showed a decrease in mean operative time during complex cases (p = 0.0008, d = 0.44), but an increase in mean operative times for noncomplex cases (p < 0.0001, d = 0.33).

CONCLUSIONS

An OC model improves chief residents' prospective entrustment leading to increased OE in cases with greater levels of operative complexity, showing a decrease in mean operative time compared to uncoached residents in certain procedures. This is the first report showing formal coaching may be a method to enhance chief resident OE.

Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.