Competency-based medical education: The spark to ignite healthcare's escape fire

High-value care is what patients deserve and what healthcare professionals should deliver. However, it is not what happens much of the time. Quality improvement master Dr. Don Berwick argued more than two decades ago that American healthcare needs an escape fire, which is a new way of seeing and acting in a crisis situation. While coined in the U.S. context, the analogy applies in other Western healthcare contexts as well. Therefore, in this paper, the authors revisit Berwick's analogy, arguing that medical education can, and should, provide the spark for such an escape fire across the globe. They assert that medical education can achieve this by fully embracing competency-based medical education (CBME) as a way to place medicine's focus on the patient. CBME targets training outcomes that prepare graduates to optimize patient care. The authors use the escape fire analogy to argue that medical educators must drop long-held approaches and tools; treat CBME implementation as an adaptive challenge rather than a technical fix; demand genuine, rich discussions and engagement about the path forward; and, above all, center the patient in all they do.

"Dismantling Fixed Time, Variable Outcome Education: Abandoning 'Ready or Not, Here they Come' is Overdue"

Two decades after competency-based medical education appeared in the lexicon of medical educators, the community continues to struggle with realizing its full potential. The implementation of the time variable, fixed outcome component has languished based on complexity compounded by resistance to change. Learners continue to transition from medical school to residency, and then practice, primarily based on time rather than having achieved the ability to meet the needs of the patient populations they will serve. Only those few who demonstrate glaring deficiencies do not graduate. The authors urge the medical education community to move from the current fixed time path of medical education toward the implementation of a true continuum of time variable, fixed outcome education, training, and deliberate practice. The latter is defined by purposeful learning, coaching, feedback, and repetition on the path to achieving and maintaining expertise. The opportunities afforded by such a time-variable, fixed outcome approach include: 1) development of a career long growth mindset, 2) ability to address evolving population health needs and careers within the context of one's practice, and 3) continual improvement of care quality and outcomes for patients on the journey towards expertise for providers.

The education passport: connecting programmatic assessment across learning and practice

Competency-based medical education (CBME) shifts us from static assessment of learning to developmental assessment for learning. However, implementation challenges associated with CBME remain a major hurdle, especially after training and into practice. The full benefit of developmental assessment for learning over time requires collaboration, cooperation, and trust among learners, regulators, and the public that transcends each individual phase. The authors introduce the concept of an "Education Passport" that provides evidence of readiness to travel across the boundaries between undergraduate medical education, graduate medical education, and the expanse of practice. The Education Passport uses programmatic assessment, a process of collecting numerous low stakes assessments from multiple sources over time, judging these data using criterion-referencing, and enhancing this with coaching and competency committees to understand, process, and accelerate growth without end. Information in the Passport is housed on a cloud-based server controlled by the student/physician over the course of training and practice. These data are mapped to various educational frameworks such Entrustable Professional Activities or milestones for ease of longitudinal performance tracking. At each stage of education and practice the student/physician grants Passport access to all entities that can provide data on performance. Database managers use learning analytics to connect and display information over time that are then used by the student/physician, their assigned or chosen coaches, and review committees to maintain or improve performance. Global information is also collected and analyzed to improve the entire system of learning and care. Developing a true continuum that embraces performance and growth will be a long-term adaptive challenge across many organizations and jurisdictions and will require coordination from regulatory and national agencies. An Education Passport could also serve as an organizing tool and will require research and high-value communication strategies to maximize public trust in the work.

Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended

PURPOSE

Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention.

METHOD

A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively.

RESULTS

After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments.

CONCLUSIONS

Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.

Medical Students' Perspectives on Entrustment Decision Making in an Entrustable Professional Activity Assessment Framework: A Secondary Data Analysis

PURPOSE

Research on how entrustment decisions are made identifies 5 influential factors (supervisor, trainee, supervisor-trainee relationship, context, task). However, this literature primarily represents the perspective of supervisors in graduate medical education and is conducted outside of an assessment framework where entrustment decisions have consequences for trainees and for patients. To complement the literature, the authors explored how medical students in a pilot program that used an entrustable professional activity (EPA) assessment framework perceived factors influencing entrustment decisions.

METHOD

The authors conducted a secondary analysis of qualitative data from program evaluation of a pilot project using an EPA assessment framework to advance students through their curriculum and into residency. Data were derived from 8 focus groups conducted over 4 years (2015-2018) with 27 students who shared their experience of learning and supervision in the project. Sensitized by the entrustment literature, the authors coded and analyzed focus group transcripts according to principles of thematic analysis.

RESULTS

Components of the trainee and supervisor-trainee relationship factors predominated students' perceptions of entrustment decisions. Students directed their own learning by asking for feedback, seeking opportunities to engage in learning, sharing limitations of their knowledge with supervisors, and pushing supervisors to recalibrate assessments when appropriate. Students' agentic actions were facilitated by longitudinal supervisor-trainee relationships wherein they felt comfortable asking for help and built confidence in patient care. Students mentioned components of other factors that influenced entrustment decisions (supervisor, clinical task, clinical context), but did so less frequently and from a nonagentic vantage point.

CONCLUSIONS

Students' perspectives on entrustment decisions can be derived from their views on learning and supervision in an EPA assessment framework. Their perspectives complement the literature by highlighting students' agentic actions to influence entrustment decisions and promotion of agentic action through practices incorporating longitudinal supervisor-trainee relationships.

Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic

Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

Continued Supervision for the Common Pediatric Subspecialty Entrustable Professional Activities May Be Needed Following Fellowship Graduation

PURPOSE

Entrustable professional activities (EPAs) are one approach to competency-based medical education (CBME), and 7 EPAs have been developed that address content relevant for all pediatric subspecialties. However, it is not known what level of supervision fellowship program directors (FPDs) deem necessary for graduation. The Subspecialty Pediatrics Investigator Network (SPIN) investigated FPD perceptions of the minimum level of supervision required for a trainee to successfully graduate.

METHOD

In 2017, SPIN surveyed all FPDs of accredited fellowships for 14 subspecialties. For each EPA, the minimum supervision level for graduation (ranging from observation only to unsupervised practice) was set such that no more than 20% of FPDs would accept a lower level.

RESULTS

The survey response rate was 82% (660/802). The minimum supervision level for graduation varied across the 7 EPAs from 2 (direct) to 4 (indirect for complex cases), with significant differences between EPAs. The percentage of FPDs desiring a lower minimum supervision level ranged from 3% to 17%. Compared with the 4 nonclinical EPAs (quality improvement, management, lead within the profession, scholarship), the 3 clinical EPAs (consultation, handover, lead a team) had higher minimum supervision graduation levels (P < .001), with less likelihood that an FPD would graduate a learner below their minimum level (P < .001).

CONCLUSIONS

Consensus among FPDs across all pediatric subspecialties demonstrates the potential need for ongoing supervision for graduates in all 7 common pediatric subspecialty EPAs after fellowship. As CBME programs are implemented, processes and infrastructure to support new graduates are important considerations for leaders.

Clarifying essential terminology in entrustment

With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

Harnessing the Potential Futures of CBME Here and Now

The COVID-19 pandemic of 2020 exposed the reactive nature of the medical education community in response to a disruption that, at one time, may have seemed preposterous. In this article, the author reflected on the impact of an unpredictable plight on a system of medical education that (1) is continuous but doesn't function as a continuum and (2) requires adaptation but is steeped in a fixed mindset and structure that resists change. As a result, innovations which were previously considered impossible, such as time variable education and training, were forced into being. Inspired by the changes brought about by the pandemic, the ensuing decade is explored through a lens of possible futures to envision a path forward based on resilience rather than reactivity.

Learner Levels of Supervision Across the Continuum of Pediatrics Training

PURPOSE

To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship.

METHOD

The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings.

RESULTS

Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency.

CONCLUSIONS

Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.

AI-ssessment: Towards Assessment As a Sociotechnical System for Learning

Two decades ago, the advent of competency-based medical education (CBME) marked a paradigm shift in assessment. Now, medical education is on the cusp of another transformation driven by advances in the field of artificial intelligence (AI). In this article, the authors explore the potential value of AI in advancing CBME and entrustable professional activities by shifting the focus of education from assessment of learning to assessment for learning. The thoughtful integration of AI technologies in observation is proposed to aid in restructuring our current system around the goal of assessment for learning by creating continuous, tight feedback loops that were not before possible. The authors argued that this personalized and less judgmental relationship between learner and machine could shift today's dominating mindset on grades and performance to one of growth and mastery learning that leads to expertise. However, because AI is neither objective nor value free, the authors stress the need for continuous co-production and evaluation of the technology with geographically and culturally diverse stakeholders to define desired behavior of the machine and assess its performance.

Facilitators and Inhibitors to Assessing Entrustable Professional Activities in Pediatric Residency

OBJECTIVE

Research on entrustable professional activities (EPAs) has focused on EPA development with little attention paid to implementation experiences. This constructivist grounded theory study sought to begin filling this gap by exploring the experiences of pediatric residency programs with implementing EPA-based assessment.

METHODS

Interviews with 19 program leader and clinical competency committee participants from 13 sites were held between January and July 2019. Participants were asked about their experiences with implementing EPA-based assessment. Data collection and analysis were iterative.

RESULTS

Participants described a range of facilitators and inhibitors that influenced their efforts to implement EPA-based assessment. These fell into 4 thematic areas: 1) alignment of EPA construct with local views of performance and assessment, 2) assessing EPAs illuminates holes in the residency curriculum, 3) clinical competency committee structure and process impacts EPA-based assessment, and 4) faculty engagement and development drives ability to assess EPAs. Areas described as facilitators by some participants were noted to be inhibitors for others. The sum of a program's facilitators and inhibitors led to more or less ability to assess EPAs on the whole. Finally, the first area functions differently from the others; it can shift the entire balance toward or away from the ability to assess EPAs overall.

CONCLUSION

This study helps fill a void in implementation evidence for EPA-based assessment through better understanding of facilitators and inhibitors to such efforts.

Entrustment Decision Making: Extending Miller's Pyramid

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid.

Implementing the Behavioral and Mental Health Entrustable Professional Activity: Insights for a Path Forward

OBJECTIVE

Mental health problems in children are growing exponentially. General pediatricians, while in a unique position to address these issues as they arise, report they lack adequate training in assessing and managing behavioral/mental health (B/MH) problems. Underscoring the importance of this area, the American Board of Pediatrics has defined B/MH as one of only 17 foundational entrustable professional activities (EPAs) for general pediatric practice. Our goal was to explore the facilitators and barriers associated with implementing and assessing the B/MH EPA among pediatric residency programs in order to identify best practices and potential solutions to common barriers.

METHODS

In this qualitative study, 18 key faculty members from 4 residency programs with 3 years' experience implementing and assessing their residents on the B/MH EPA were purposively sampled. Semistructured interviews were conducted with each participant, and interviews were analyzed utilizing a thematic analysis.

RESULTS

Five themes were defined in the thematic analysis 1) B/MH training: who's responsible? 2) local context can serve as a barrier or facilitator, 3) B/MH may require longitudinal, integrated, and multidisciplinary training, 4) B/MH specialists: indispensable, yet a hurdle?, and 5) resident and faculty confidence and skill impact B/MH training.

CONCLUSIONS

The need for robust training to prepare pediatric residency graduates to meet the needs of patients with B/MH problems has never been greater. This study provides important insights about gaps in B/MH training. These should inform future directions focused on addressing this need.

Use of Resident-Sensitive Quality Measure Data in Entrustment Decision Making: A Qualitative Study of Clinical Competency Committee Members at One Pediatric Residency

PURPOSE

Resident-sensitive quality measures (RSQMs) are quality measures that are likely performed by an individual resident and are important to care quality for a given illness of interest. This study sought to explore how individual clinical competency committee (CCC) members interpret, use, and prioritize RSQMs alongside traditional assessment data when making a summative entrustment decision.

METHOD

In this constructivist grounded theory study, 19 members of the pediatric residency CCC at Cincinnati Children's Hospital Medical Center were purposively and theoretically sampled between February and July 2019. Participants were provided a deidentified resident assessment portfolio with traditional assessment data (milestone and/or entrustable professional activity ratings as well as narrative comments from 5 rotations) and RSQM performance data for 3 acute, common diagnoses in the pediatric emergency department (asthma, bronchiolitis, and closed head injury) from the emergency medicine rotation. Data collection consisted of 2 phases: (1) observation and think out loud while participants reviewed the portfolio and (2) semistructured interviews to probe participants' reviews. Analysis moved from close readings to coding and theme development, followed by the creation of a model illustrating theme interaction. Data collection and analysis were iterative.

RESULTS

Five dimensions for how participants interpret, use, and prioritize RSQMs were identified: (1) ability to orient to RSQMs: confusing to self-explanatory, (2) propensity to use RSQMs: reluctant to enthusiastic, (3) RSQM interpretation: requires contextualization to self-evident, (4) RSQMs for assessment decisions: not sticky to sticky, and (5) expectations for residents: potentially unfair to fair to use RSQMs. The interactions among these dimensions generated 3 RSQM data user profiles: eager incorporation, willing incorporation, and disinclined incorporation.

CONCLUSIONS

Participants used RSQMs to varying extents in their review of resident data and found such data helpful to varying degrees, supporting the inclusion of RSQMs as resident assessment data for CCC review.

Reported Pediatrics Milestones (Mostly) Measure Program, Not Learner Performance

PURPOSE

Semiannually, U.S. pediatrics residency programs report resident milestone levels to the Accreditation Council for Graduate Medical Education (ACGME). The Pediatrics Milestones Assessment Collaborative (PMAC, consisting of the National Board of Medical Examiners, American Board of Pediatrics, and Association of Pediatric Program Directors) developed workplace-based assessments of 2 inferences: readiness to serve as an intern with a supervisor present (D1) and readiness to care for patients with a supervisor nearby in the pediatric inpatient setting (D2). The authors compared learner and program variance in PMAC scores with ACGME milestones.

METHOD

The authors examined sources of variance in PMAC scores and milestones between November 2015 and May 2017 of 181 interns at 8 U.S. pediatrics residency programs using random effects models with program, competency, learner, and program × competency components.

RESULTS

Program-related milestone variance was substantial (54% D1, 68% D2), both in comparison to learner milestone variance (22% D1, 14% D2) and program variance in the PMAC scores (12% D1, 10% D2). In contrast, learner variance represented 44% (D1) or 26% (D2) of variance in PMAC scores. Within programs, PMAC scores were positively correlated with milestones for all but one competency.

CONCLUSIONS

PMAC assessments provided scores with little program-specific variance and were more sensitive to differences in learners within programs compared with milestones. Milestones reflected greater differences by program than by learner. This may represent program-based differences in intern performance or in use of milestones as a reporting scale. Comparing individual learner milestones without adjusting for programs is problematic.

Narrative Performance Level Assignments at Initial Entrustment and Graduation: Integrating EPAs and Milestones to Improve Learner Assessment

PURPOSE

To determine which narrative performance level for each general pediatrics entrustable professional activity (EPA) reflects the minimum level clinical competency committees (CCCs) felt should be associated with graduation as well as initial entrustment and compare expected narrative performance levels (ENPLs) for each EPA with actual narrative performance levels (ANPLs) assigned to residents at initial entrustment.

METHOD

A series of 5 narratives, corresponding to the 5 milestone performance levels, were developed for each of the 17 general pediatrics EPAs. In academic year (AY) 2015-2016, the CCCs at 22 Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network member sites reported ENPLs for initial entrustment and at time of graduation. From AYs 2015-2016 to 2017-2018, programs reported ANPLs for initial entrustment decisions. ENPLs and ANPLs were compared using a logistic mixed effects model.

RESULTS

ENPLs for graduation and entrustment were most often level 3 (competent) followed by level 4 (proficient). For 8 EPAs, the ENPLs for graduation and entrustment were the same. For the remaining 9, some programs would entrust residents before graduation or graduate them before entrusting them. There were 4,266 supervision level reports for initial entrustment for which an ANPL was provided. ANPLs that were lower than the ENPLs were significantly more likely to be assigned to the medical home-well child (OR = 0.39; 95% CI: 0.26-0.57), transition to adult care (OR = 0.43; 95% CI: 0.19-0.95), behavioral or mental health (OR = 0.36; 95% CI: 0.18-0.71), make referrals (OR = 0.31; 95% CI: 0.17-0.55), lead a team (OR = 0.34; 95% CI: 0.22-0.52), and handovers (OR = 0.18; 95% CI: 0.09-0.36) EPAs.

CONCLUSIONS

CCCs reported lower ENPLs for graduation than for entrustment for 5 EPAs, possibly indicating curricular gaps that milestones and EPAs could help identify.

Resident-Sensitive Quality Measures in the Pediatric Emergency Department: Exploring Relationships With Supervisor Entrustment and Patient Acuity and Complexity

PURPOSE

This study explores the associations between resident-sensitive quality measures (RSQMs) and supervisor entrustment as well as between RSQMs and patient acuity and complexity for encounters in the pediatric emergency department (PED) in which residents are caring for patients.

METHOD

Pediatric residents rotating through Cincinnati Children's Hospital Medical Center PED as well as supervising pediatric emergency medicine faculty and fellows were recruited during the 2017-2018 academic year for the purpose of collecting the following data from the residents' patient encounters for 3 illnesses (acute asthma exacerbation, bronchiolitis, and closed head injury [CHI]): supervisor entrustment decision rating, RSQMs relevant to the care provided, and supervisor patient acuity and complexity ratings. To measure the association of RSQM composite scores with the other variables of interest, mixed models were used.

RESULTS

A total of 83 residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with CHI. Entrustment decision ratings were positively associated with asthma RSQM composite scores (beta coefficient = 0.03; P < .001). There was no significant association between RSQM composite scores and entrustment decision ratings for bronchiolitis or CHI. RSQM composite scores were significantly higher when acuity was also higher and significantly lower when acuity was also lower for both asthma (P < .001) and bronchiolitis (P = .01). However, RSQM composite scores were almost identical between levels of acuity for CHI (P = .94). There were no significant differences in RSQM composite scores when complexity varied.

CONCLUSION

This study found limited associations between RSQM composite scores and entrustment decision ratings but offers insight into how RSQMs could be used for the purposes of resident assessment and feedback.

Initial Implementation of Resident-Sensitive Quality Measures in the Pediatric Emergency Department: A Wide Range of Performance

PURPOSE

A lack of quality measures aligned with residents' work led to the development of resident-sensitive quality measures (RSQMs). This study sought to describe how often residents complete RSQMs, both individually and collectively, when they are implemented in the clinical environment.

METHOD

During academic year 2017-2018, categorical pediatric residents in the Cincinnati Children's Hospital Medical Center pediatric emergency department were assessed using RSQMs for acute asthma exacerbation (21 RSQMs), bronchiolitis (23 RSQMs), and closed head injury (19 RSQMs). Following eligible patient encounters, all individual RSQMs for the illnesses of interest were extracted from the health record. Frequencies of 3 performance classifications (opportunity and met, opportunity and not met, or no opportunity) were detailed for each RSQM. A composite score for each encounter was calculated by determining the proportion of individual RSQMs performed out of the total possible RSQMs that could have been performed.

RESULTS

Eighty-three residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with closed head injury during the study period. Residents had the opportunity to meet the RSQMs in most encounters, but exceptions existed. There was a wide range in the frequency of residents meeting RSQMs in encounters in which the opportunity existed. One closed head injury measure was met in all encounters in which the opportunity existed. Across illnesses, some RSQMs were met in almost all encounters, while others were met in far fewer encounters. RSQM composite scores demonstrated significant range and variation as well-asthma: mean = 0.81 (standard deviation [SD] = 0.11) and range = 0.47-1.00, bronchiolitis: mean = 0.62 (SD = 0.12) and range = 0.35-0.91, and closed head injury: mean = 0.63 (SD = 0.10) and range = 0.44-0.89.

CONCLUSIONS

Individually and collectively, RSQMs can distinguish variations in the tasks residents perform across patient encounters.

The Power of Contribution and Attribution in Assessing Educational Outcomes for Individuals, Teams, and Programs

Recent discussions have brought attention to the utility of contribution analysis for evaluating the effectiveness and outcomes of medical education programs, especially for complex initiatives such as competency-based medical education. Contribution analysis focuses on the extent to which different entities contribute to an outcome. Given that health care is provided by teams, contribution analysis is well suited to evaluating the outcomes of care delivery. Furthermore, contribution analysis plays an important role in analyzing program- and system-level outcomes that inform program evaluation and program-level improvements for the future. Equally important in health care, however, is the role of the individual. In the overall contribution of a team to an outcome, some aspects of this outcome can be attributed to individual team members. For example, a recently discharged patient with an unplanned return to the emergency department to seek care may not have understood the discharge instructions given by the nurse or may not have received any discharge guidance from the resident physician. In this example, if it is the nurse's responsibility to provide discharge instructions, that activity is attributed to him or her. This and other activities attributed to different individuals (e.g., nurse, resident) combine to contribute to the outcome for the patient. Determining how to tease out such attributions is important for several reasons. First, it is physicians, not teams, that graduate and are granted certification and credentials for medical practice. Second, incentive-based payment models focus on the quality of care provided by an individual. Third, an individual can use data about his or her performance on the team to help drive personal improvement. In this article, the authors explored how attribution and contribution analyses can be used in a complimentary fashion to discern which outcomes can and should be attributed to individuals, which to teams, and which to programs.

Conditions Influencing Collaboration Among the Primary Care Disciplines as They Prepare the Future Primary Care Physician Workforce

BACKGROUND AND OBJECTIVES

Much can be gained by the three primary care disciplines collaborating on efforts to transform residency training toward interprofessional collaborative practice. We describe findings from a study designed to align primary care disciplines toward implementing interprofessional education.

METHODS

In this mixed methods study, we included faculty, residents and other interprofessional learners in family medicine, internal medicine, and pediatrics from nine institutions across the United States. We administered a web-based survey in April/May of 2018 and used qualitative analyses of field notes to study resident exposure to team-based care during training, estimates of career choice in programs that are innovating, and supportive and challenging conditions that influence collaboration among the three disciplines. Complete data capture was attained for 96.3% of participants.

RESULTS

Among family medicine resident graduates, an estimated 87.1% chose to go into primary care compared to 12.4% of internal medicine, and 36.5% of pediatric resident graduates. Qualitative themes found to positively influence cross-disciplinary collaboration included relationship development, communication of shared goals, alignment with health system/other institutional initiatives, and professional identity as primary care physicians. Challenges included expressed concerns by participants that by working together, the disciplines would experience a loss of identity and would be indistinguishable from one another. Another qualitative finding was that overwhelming stressors plague primary care training programs in the current health care climate-a great concern. These include competing demands, disruptive transitions, and lack of resources.

CONCLUSIONS

Uniting the primary care disciplines toward educational and clinical transformation toward interprofessional collaborative practice is challenging to accomplish.

Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities

IMPORTANCE

Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence.

OBJECTIVE

To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data.

DESIGN, SETTING, AND PARTICIPANTS

Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018.

INTERVENTIONS

Longitudinal, prospective assessment using EPAs.

MAIN OUTCOMES AND MEASURES

Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA.

RESULTS

Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician).

CONCLUSIONS AND RELEVANCE

This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Agreement of Program Directors With Clinical Competency Committees for Fellow Entrustment

OBJECTIVES

Fellowship program directors (FPD) and Clinical Competency Committees (CCCs) both assess fellow performance. We examined the association of entrustment levels determined by the FPD with those of the CCC for 6 common pediatric subspecialty entrustable professional activities (EPAs), hypothesizing there would be strong correlation and minimal bias between these raters.

METHODS

The FPDs and CCCs separately assigned a level of supervision to each of their fellows for 6 common pediatric subspecialty EPAs. For each EPA, we determined the correlation between FPD and CCC assessments and calculated bias as CCC minus FPD values for when the FPD was or was not a member of the CCC. In addition, we examined the effect of program size, FPD understanding of EPAs, and subspecialty on the correlations. Data were obtained in fall 2014 and spring 2015.

RESULTS

A total of 1040 fellows were assessed in the fall and 1048 in the spring. In both periods and for each EPA, there was a strong correlation between FPD and CCC supervision levels (P < .001). The correlation was somewhat lower when the FPD was not a CCC member (P < .001). Overall bias in both periods was small.

CONCLUSIONS

The correlation between FPD and CCC assignment of EPA supervision levels is strong. Although slightly weaker when the FPD is not a CCC member, bias is small, so this is likely unimportant in determining fellow entrustment level. The similar performance ratings of FPDs and CCCs support the validity argument for EPAs as competency-based assessment tools.

A Feasibility Study to Attribute Patients to Primary Interns on Inpatient Ward Teams Using Electronic Health Record Data

PURPOSE

To inform graduate medical education (GME) outcomes at the individual resident level, this study sought a method for attributing care for individual patients to individual interns based on "footprints" in the electronic health record (EHR).

METHOD

Primary interns caring for patients on an internal medicine inpatient service were recorded daily by five attending physicians of record at University of Cincinnati Medical Center in August 2017 and January 2018. These records were considered gold standard identification of primary interns. The following EHR variables were explored to determine representation of primary intern involvement in care: postgraduate year, progress note author, discharge summary author, physician order placement, and logging clicks in the patient record. These variables were turned into quantitative attributes (e.g., progress note author: yes/no), and informative attributes were selected and modeled using a decision tree algorithm.

RESULTS

A total of 1,511 access records were generated; 116 were marked as having a primary intern assigned. All variables except discharge summary author displayed at least some level of importance in the models. The best model achieved 78.95% sensitivity, 97.61% specificity, and an area under the receiver-operator curve of approximately 91%.

CONCLUSIONS

This study successfully predicted primary interns caring for patients on inpatient teams using EHR data with excellent model performance. This provides a foundation for attributing patients to primary interns for the purposes of determining patient diagnoses and complexity the interns see as well as supporting continuous quality improvement efforts in GME.

Envisioning a True Continuum of Competency-Based Medical Education, Training, and Practice

The existing structure of physician education has developed in siloed stages, with consecutive degrees and certifications and progressively longer training programs. As further fragmentation of health care and training systems will not improve the quality of care and education, the authors argue that a new vision of education, training, and practice as a continuum is needed.They advocate for a model of competency-based medical education that merges with competency-based medical practice. In this system, education and training will result in individual, dynamic portfolios of valid entrustable professional activities (EPAs) for which physicians are certified. Physicians can maintain and renew that entrustment as long as the EPAs remain within their scope of practice. Entrustment occurs initially during training but is then granted for new activities as physicians' careers evolve.This model accounts for the need to keep pace with changes in population health needs and expectations of competence over time. It de-emphasizes the divides between the stages of training and views the continuum from undergraduate medical education until retirement as a whole. Key obligations of self-regulating medical professionals include both the reception and the provision of supervision from and for others, respectively. Learning must be embedded in practice to address expectations regarding new knowledge and skills as they evolve with scientific and technological advances.Entrusting physicians to deliver effective and safe care, based on their performance of the requisite EPAs without supervision, should ensure that they provide high-value, quality care to patients.

Crossing the Gap: Using Competency-Based Assessment to Determine Whether Learners Are Ready for the Undergraduate-to-Graduate Transition

In 2011, the Education in Pediatrics Across the Continuum (EPAC) Study Group recruited four medical schools (University of California, San Francisco; University of Colorado; University of Minnesota; and University of Utah) and their associated pediatrics clerkship and residency program directors to be part of a consortium to pilot a model designed to advance learners from undergraduate medical education (UME) to graduate medical education (GME) and then to fellowship or practice based on competence rather than time spent in training. The central design features of this pilot included predetermined expectations of performance and transition criteria to ensure readiness to progress from UME to GME, using the Core Entrustable Professional Activities for Entering Residency (Core EPAs) as a common assessment framework. Using this framework, each site team (which included, but was not limited to, the EPAC course, pediatric clerkship, and pediatric residency program directors) monitored learners' progress, with the site's clinical competency committee marking the point of readiness to transition from UME to GME (i.e., the attainment of supervision level 3a). Two of the sites implemented time-variable transition from UME to GME, based on when a learner met the performance expectations and transition criteria. In this Article, the authors describe each of the four sites' implementation of Core EPA assessment and their approach to gathering the data necessary to determine readiness for transition. They conclude by offering recommendations and lessons learned from the pilot's first seven years of development, adaptation, and implementation of assessment strategies across the sites, and discussing next steps.

Developing Resident-Sensitive Quality Measures: Engaging Stakeholders to Inform Next Steps

OBJECTIVE

Despite the need for quality measures relevant to the work residents complete, few attempts have been made to address this gap. Resident-sensitive quality measures (RSQMs) can help fill this void. This study engaged resident and supervisor stakeholders to develop and inform next steps in creating such measures.

METHODS

Two separate nominal group techniques (NGTs), one with residents and one with faculty and fellow supervisors, were used to generate RSQMs for 3 specific illnesses (asthma, bronchiolitis, and closed head injury) as well as general care for the pediatric emergency department. Two separate Delphi processes were then used to prioritize identified RSQMs. The measures produced by each group were compared side by side, illuminating similarities and differences that were explored through focus groups with residents and supervisors. These focus groups also probed future settings in which to develop RSQMs.

RESULTS

In the NGT and Delphi groups, residents and supervisors placed considerable focus on measures in 3 areas across the illnesses of interest: 1) appropriate medication dosing, 2) documentation, and 3) information provided at patient discharge. Focus groups highlighted hospital medicine and general pediatrics as priority areas for developing future RSQMs but also noted contextual variables that influence the application of similar measures in different settings. Residents and supervisors had both similar as well as unique insights into developing RSQMs.

CONCLUSIONS

This study continues to pave the path forward in developing future RSQMs by exploring specific settings, measures, and stakeholders to consider when undertaking this work.

MOCA-Peds: Development of a New Assessment of Medical Knowledge for Continuing Certification

The American Board of Pediatrics (ABP) certifies that general and subspecialty pediatricians meet standards of excellence established by their peers. Certification helps demonstrate that a general pediatrician or pediatric subspecialist has successfully completed accredited training and fulfills continuous certification requirements (Maintenance of Certification [MOC]). One current component of the MOC program is a closed-book examination administered at a secure testing center (ie, the MOC Part 3 examination). In this article, we describe the development of an alternative to this examination termed the "Maintenance of Certification Assessment for Pediatrics" (MOCA-Peds) during 2015-2016. MOCA-Peds was conceptualized as an online, summative (ie, pass/fail), continuous assessment of a pediatrician's knowledge that would also promote learning. The system would consist of a set number of multiple-choice questions delivered each quarter, with immediate feedback on questions, rationales clarifying correct and incorrect answers, references for further learning, and peer benchmarking. Questions would be delivered quarterly and taken at any time within the quarter in a setting with Internet connectivity and on any device. As part of the development process in 2015-2016, the ABP actively recruited pediatricians to serve as members of a yearlong user panel or single-session focus groups. Refinements to MOCA-Peds were made on the basis of their feedback. MOCA-Peds is being actively piloted with pediatricians in 2017-2018. The ABP anticipates an expected launch in January 2019 of MOCA-Peds for General Pediatrics, Pediatric Gastroenterology, Child Abuse, and Pediatric Infectious Diseases with launch dates for the remaining pediatric subspecialties between 2020 and 2022.

Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth From Ten Pediatrics Residency Programs

PURPOSE

This study investigates the impact of incorporating observer-reported workload into workplace-based assessment (WBA) scores on (1) psychometric characteristics of WBA scores and (2) measuring changes in performance over time using workload-unadjusted versus workload-adjusted scores.

METHOD

Structured clinical observations and multisource feedback instruments were used to collect WBA data from first-year pediatrics residents at 10 residency programs between July 2016 and June 2017. Observers completed items in 8 subcompetencies associated with Pediatrics Milestones. Faculty and resident observers assessed workload using a sliding scale ranging from low to high; all item scores were rescaled to a 1-5 scale to facilitate analysis and interpretation. Workload-adjusted WBA scores were calculated at the item level using three different approaches, and aggregated for analysis at the competency level. Mixed-effects regression models were used to estimate variance components. Longitudinal growth curve analyses examined patterns of developmental score change over time.

RESULTS

On average, participating residents (n = 252) were assessed 5.32 times (standard deviation = 3.79) by different raters during the data collection period. Adjusting for workload yielded better discrimination of learner performance, and higher reliability, reducing measurement error by 28%. Projections in reliability indicated needing up to twice the number of raters when workload-unadjusted scores were used. Longitudinal analysis showed an increase in scores over time, with significant interaction between workload and time; workload also increased significantly over time.

CONCLUSIONS

Incorporating a measure of observer-reported workload could improve the measurement properties and the ability to interpret WBA scores.

Developing Resident-Sensitive Quality Measures: A Model From Pediatric Emergency Medicine

PURPOSE

To begin closing the gap with respect to quality measures available for use among residents, the authors sought to identify and develop resident-sensitive quality measures (RSQMs) for use in the pediatric emergency department (PED) setting.

METHOD

In May 2016, the authors reviewed National Quality Measures Clearinghouse (NQMC) measures to identify resident-sensitive measures. To create additional measures focused on common, acute illnesses (acute asthma exacerbation, bronchiolitis, closed head injury [CHI]) in the PED, the authors used a nominal group technique (NGT) and Delphi process from September to December 2016. To achieve a local focus for developing these measures, all NGT and Delphi participants were from Cincinnati Children's Hospital Medical Center. Delphi participants rated measures developed through the NGT in two areas: importance of measure to quality care and likelihood that measure represents the work of a resident.

RESULTS

The review of NQMC measures identified 28 of 183 as being potentially resident sensitive. The NGT produced 67 measures for asthma, 46 for bronchiolitis, and 48 for CHI. These were used in the first round of the Delphi process. After two rounds, 18 measures for asthma, 21 for bronchiolitis, and 21 for CHI met automatic inclusion criteria. In round three, participants categorized the potential final measures by their top 10 and next 5.

CONCLUSIONS

This study describes a template for identifying and developing RSQMs that may promote high-quality care delivery during and following training. Next steps should include implementing and seeking validity evidence for the locally developed measures.

Strengthening the Connection of Medical Education to the Vision of Improving Child Health

This is the text version of Dr Carol Caraccio’s acceptance speech upon receiving the 2017 Joseph W. St. Geme, Jr, Leadership Award.

Education in Pediatrics Across the Continuum (EPAC): First Steps Toward Realizing the Dream of Competency-Based Education

The Education in Pediatrics Across the Continuum (EPAC) Study Group is developing the first competency-based, time-variable progression from undergraduate medical education (UME) to graduate medical education (GME) in the history of medical education in the United States. EPAC, an innovation project sponsored by the Association of American Medical Colleges and supported by the Josiah Macy Jr. Foundation, was developed through a collaboration between five medical schools and multiple professional organizations with an interest in undergraduate and graduate medical education. The planning and implementation process demanded cooperatively addressing practical barriers such as education requirements for licensure and developing approaches to learner assessment that provided meaningful information about competency. Each participating school now has at least three cohorts of learners participating, and the program is transitioning its first cohort of students from UME to GME based on achievement of predetermined competencies that allow this transition. Members of the first cohort of learners in this program have begun their pediatric residency training at different times beginning in late 2016, confirming the feasibility of competency-based advancement from UME to GME in pediatrics. Although there is still much to learn about the outcomes of EPAC learners' professional development in residency training and beyond, EPAC has defined an operational approach to a different path through medical school and into residency training, based on the attainment of competence.

A Lack of Continuity in Education, Training, and Practice Violates the "Do No Harm" Principle

The paradigm shift to competency-based medical education (CBME) is under way, but incomplete implementation is blunting the potential impact on learning and patient outcomes. The fundamental principles of CBME call for standardizing outcomes addressing population health needs, then allowing time-variable progression to achieving them. Operationalizing CBME principles requires continuity within and across phases of the education, training, and practice continuum. However, the piecemeal origin of the phases of the "continuum" has resulted in a sequence of undergraduate to graduate medical education to practice that may be continuous temporally but bears none of the integration of a true continuum.With these timed interruptions during phase transitions, learning is not reinforced because of a failure to integrate experiences. Brief block rotations for learners and ever-shorter supervisory assignments for faculty preclude the development of relationships. Without these relationships, feedback falls on deaf ears. Block rotations also disrupt learners' relationships with patients. The harms resulting from such a system include decreases in patient satisfaction with their care and learner satisfaction with their work. Learners in this block system also demonstrate an erosion of empathy compared with those in innovative longitudinal training models. In addition, higher patient mortality during intern transitions has been demonstrated.The current medical education system is violating the first principle of medicine: "Do no harm." Full implementation of competency-based, time-variable education and training, with fixed outcomes aligned with population health needs, continuity in learning and relationships, and support from a developmental program of assessment, holds great potential to stop this harm.

Toward a shared language for competency-based medical education

The paradigm shift brought about by the advent of competency-based medical education (CBME) can be characterized as an adaptive change. Currently, its development and implementation suffer from the lack of a lingua franca. A shared language is needed to support collaboration and dissemination across the world community of medical educators. The International CBME Collaborators held a second summit in 2013 to explore this and other contemporary CBME issues. We present the resulting International CBME Collaborator's glossary of CBME terms. Particular attention is given to the terms competency, entrustable professional activity (EPA), and milestone and their interrelationships. Medical education scholars and enthusiasts of the competency-based approach are encouraged to adopt these terms and definitions, although no doubt the vocabulary of CBME will continue to evolve.

Overarching challenges to the implementation of competency-based medical education

Medical education is under increasing pressure to more effectively prepare physicians to meet the needs of patients and populations. With its emphasis on individual, programmatic, and institutional outcomes, competency-based medical education (CBME) has the potential to realign medical education with this societal expectation. Implementing CBME, however, comes with significant challenges. This manuscript describes four overarching challenges that must be confronted by medical educators worldwide in the implementation of CBME: (1) the need to align all regulatory stakeholders in order to facilitate the optimization of training programs and learning environments so that they support competency-based progression; (2) the purposeful integration of efforts to redesign both medical education and the delivery of clinical care; (3) the need to establish expected outcomes for individuals, programs, training institutions, and health care systems so that performance can be measured; and (4) the need to establish a culture of mutual accountability for the achievement of these defined outcomes. In overcoming these challenges, medical educators, leaders, and policy-makers will need to seek collaborative approaches to common problems and to learn from innovators who have already successfully made the transition to CBME.

Core principles of assessment in competency-based medical education

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.

Building a Framework of Entrustable Professional Activities, Supported by Competencies and Milestones, to Bridge the Educational Continuum

The transition to competency-based medical education (CBME) and adoption of the foundational domains of competence by the Accreditation Council for Graduate Medical Education, Association of American Medical Colleges (AAMC), and American Board of Medical Specialties' certification and maintenance of certification (MOC) programs provided an unprecedented opportunity for the pediatrics community to create a model of learning and assessment across the continuum. Two frameworks for assessment in CBME have been promoted: (1) entrustable professional activities (EPAs) and (2) milestones that define a developmental trajectory for individual competencies. EPAs are observable and measureable units of work that can be mapped to competencies and milestones critical to performing them safely and effectively.The pediatrics community integrated the two frameworks to create a potential pathway of learning and assessment across the continuum from undergraduate medical education (UME) to graduate medical education (GME) and from GME to practice. The authors briefly describe the evolution of the Pediatrics Milestone Project and the process for identifying EPAs for the specialty and subspecialties of pediatrics. The method of integrating EPAs with competencies and milestones through a mapping process is discussed, and an example is provided. The authors illustrate the alignment of the AAMC's Core EPAs for Entering Residency with the general pediatrics EPAs and, in turn, the alignment of the latter with the subspecialty EPAs, thus helping build the bridge between UME and GME. The authors propose how assessment in GME, based on EPAs and milestones, can guide MOC to complete the bridge across the education continuum.

Toward Defining the Foundation of the MD Degree: Core Entrustable Professional Activities for Entering Residency

Currently, no standard defines the clinical skills that medical students must demonstrate upon graduation. The Liaison Committee on Medical Education bases its standards on required subject matter and student experiences rather than on observable educational outcomes. The absence of such established outcomes for MD graduates contributes to the gap between program directors' expectations and new residents' performance.In response, in 2013, the Association of American Medical Colleges convened a panel of experts from undergraduate and graduate medical education to define the professional activities that every resident should be able to do without direct supervision on day one of residency, regardless of specialty. Using a conceptual framework of entrustable professional activities (EPAs), this Drafting Panel reviewed the literature and sought input from the health professions education community. The result of this process was the publication of 13 core EPAs for entering residency in 2014. Each EPA includes a description, a list of key functions, links to critical competencies and milestones, and narrative descriptions of expected behaviors and clinical vignettes for both novice learners and learners ready for entrustment.The medical education community has already begun to develop the curricula, assessment tools, faculty development resources, and pathways to entrustment for each of the 13 EPAs. Adoption of these core EPAs could significantly narrow the gap between program directors' expectations and new residents' performance, enhancing patient safety and increasing residents', educators', and patients' confidence in the care these learners provide in the first months of their residency training.

A Model for Catalyzing Educational and Clinical Transformation in Primary Care: Outcomes From a Partnership Among Family Medicine, Internal Medicine, and Pediatrics

PURPOSE

To report findings from a national effort initiated by three primary care certifying boards to catalyze change in primary care training.

METHOD

In this mixed-method pilot study (2012-2014), 36 faculty in 12 primary care residencies (family medicine, internal medicine, pediatrics) from four institutions participated in a professional development program designed to prepare faculty to accelerate change in primary care residency training by uniting them in a common mission to create effective ambulatory clinical learning environments. Surveys administered at baseline and 12 months after initial training measured changes in faculty members' confidence and skills, continuity clinics, and residency training programs. Feasibility evaluation involved assessing participation. The authors compared quantitative data using Wilcoxon signed-rank and Bhapkar tests. Observational field notes underwent narrative analysis.

RESULTS

Most participants attended two in-person training sessions (92% and 72%, respectively). Between baseline and 12 months, faculty members' confidence in leadership improved significantly for 15/19 (79%) variables assessed; their self-assessed skills improved significantly for 21/22 (95%) competencies. Two medical home domains ("Continuity of Care," "Support/Care Coordination") improved significantly (P < .05) between the two time periods. Analyses of qualitative data revealed that interdisciplinary learning communities formed during the program and served to catalyze transformational change.

CONCLUSIONS

Results suggest that improvements in faculty perceptions of confidence and skills occurred and that the creation of interdisciplinary learning communities catalyzed transformation. Lengthening the intervention period, engaging other professions involved in training the primary care workforce, and a more discriminating evaluation design are needed to scale this model nationally.

Advancing Competency-Based Medical Education: A Charter for Clinician-Educators

The International Competency-Based Medical Education (ICBME) Collaborators have been working since 2009 to promote understanding of competency-based medical education (CBME) and accelerate its uptake worldwide. This article presents a charter, supported by a literature-based rationale, which is meant to provide a shared mental model of CBME that will serve as a path forward in its widespread implementation.At a 2013 summit, the ICBME Collaborators laid the groundwork for this charter. Here, the fundamental principles of CBME and professional responsibilities of medical educators in its implementation process are described. The authors outline three fundamental principles: (1) Medical education must be based on the health needs of the populations served; (2) the primary focus of education and training should be the desired outcomes for learners rather than the structure and process of the educational system; and (3) the formation of a physician should be seamless across the continuum of education, training, and practice.Building on these principles, medical educators must demonstrate commitment to teaching, assessing, and role modeling the range of identified competencies. In the clinical setting, they must provide supervision that balances patient safety with the professional development of learners, being transparent with stakeholders about level of supervision needed. They must use effective and efficient assessment strategies and tools for basing transition decisions on competence rather than time in training, empowering learners to be active participants in their learning and assessment. Finally, advancing CBME requires program evaluation and research, faculty development, and a collaborative approach to realize its full potential.

The Creation of Standard-Setting Videos to Support Faculty Observations of Learner Performance and Entrustment Decisions

Entrustable professional activities (EPAs) provide a framework to standardize medical education outcomes and advance competency-based assessment. Direct observation of performance plays a central role in entrustment decisions; however, data obtained from these observations are often insufficient to draw valid high-stakes conclusions. One approach to enhancing the reliability and validity of these assessments is to create videos that establish performance standards to train faculty observers. Little is known about how to create videos that can serve as standards for assessment of EPAs.The authors report their experience developing videos that represent five levels of performance for an EPA for patient handoffs. The authors describe a process that begins with mapping the EPA to the critical competencies needed to make an entrustment decision. Each competency is then defined by five milestones (behavioral descriptors of performance at five advancing levels). Integration of the milestones at each level across competencies enabled the creation of clinical vignettes that were converted into video scripts and ultimately videos. Each video represented a performance standard from novice to expert. The process included multiple assessments by experts to guide iterative improvements, provide evidence of content validity, and ensure that the authors successfully translated behavioral descriptions and vignettes into videos that represented the intended performance level for a learner. The steps outlined are generalizable to other EPAs, serving as a guide for others to develop videos to train faculty. This process provides the level of content validity evidence necessary to support using videos as standards for high-stakes entrustment decisions.

Driving Care Quality: Aligning Trainee Assessment and Supervision Through Practical Application of Entrustable Professional Activities, Competencies, and Milestones

To address the long-standing challenge of meaningful trainee assessment, the authors reviewed and expanded on the Accountable Assessment for Quality Care and Supervision (AAQCS) equation. The equation proposes that care quality is the product of the interaction between trainee performance (measured by workplace assessment) and supervision (required level of intervention to ensure care quality) in the context of the environment where the care occurs: Trainee performance × Appropriate supervision = Safe, effective patient-centered care. Assessing trainee performance and matching that performance to "appropriate" supervision, however, is fraught with challenges. The authors suggest a unifying framework that integrates entrustable professional activities (EPAs), competencies, and milestones to inform trainee assessment and supervision, thereby enabling the practical application of the AAQCS equation in the workplace. Because the unit of measure for an EPA is the outcome of whether the trainee can safely and effectively perform the professional activity without supervision, the proposed unifying framework directly aligns with the dependent variable in the AAQCS equation: care quality.The value of applying a unifying framework that integrates EPAs, competencies, and milestones to the AAQCS equation in the clinical learning environment lies in its ability to provide supervisors with a shared mental model of performance expectations for trainees, reducing unwanted variability and improving assessment accuracy; guidance for aligning performance milestones of trainees with the needed level of supervisor intervention to ensure care quality; and substrate for specific feedback to improve the trainee's professional development as a way to ensure future care quality.

Transforming primary care residency training: a collaborative faculty development initiative among family medicine, internal medicine, and pediatric residencies

PROBLEM

The scope and scale of developments in health care redesign have not been sufficiently adopted in primary care residency programs.

APPROACH

The interdisciplinary Primary Care Faculty Development Initiative was created to teach faculty how to accelerate revisions in primary care residency training. The program focused on skill development in teamwork, change management, leadership, population management, clinical microsystems, and competency assessment. The 2013 pilot program involved 36 family medicine, internal medicine, and pediatric faculty members from 12 residencies in four locations.

OUTCOMES

The percentage of participants rating intention to implement what was learned as "very likely to" or "absolutely will" was 16/32 (50%) for leadership, 24/33 (72.7%) for change management, 23/33 (69.7%) for systems thinking, 25/32 (75.8%) for population management, 28/33 (84.9%) for teamwork, 29/33 (87.8%) for competency assessment, and 30/31 (96.7%) for patient centeredness.Content analysis revealed five key themes: leadership skills are key drivers of change, but program faculty face big challenges in changing culture and engaging stakeholders; access to data from electronic health records for population management is a universal challenge; readiness to change varies among the three disciplines and among residencies within each discipline; focusing on patients and their needs galvanizes collaborative efforts across disciplines and within residencies; and collaboration among disciplines to develop and use shared measures of residency programs and learner outcomes can guide and inspire program changes and urgently needed educational research.

NEXT STEPS

Revise and reevaluate this rapidly evolving program toward widespread engagement with family medicine, internal medicine, and pediatric residencies.

From theory to practice: making entrustable professional activities come to life in the context of milestones

Entrustable professional activities (EPAs) are gaining traction across the globe as a practical way to teach and assess competencies in the clinical setting. Full-scale implementation, though, has only taken place in obstetrics-gynecology in the Netherlands and in psychiatry in Australia and New Zealand. As with any conceptual framework, implementation in different contexts will require adaptations. For example, implementation in the United States will need to incorporate the Accreditation Council for Graduate Medical Education's competencies and the recently completed milestones for each of the specialties. In this issue, an article by Aylward and colleagues describes the process for implementing a handoff communication EPA, using milestones as the basis for the assessment tool. The explicit linkage of the milestones with the EPA assessment allows a more definitive "picture" of the learner to emerge at each advancing level of performance of the EPA. This "picture" can be shared with those directly observing the learner and thus provides a potential model for a more reliable assessment of learners performing EPAs and perhaps a more consistent approach to entrustment decisions. The authors hope that Aylward and colleagues' article will be one of many that aim to help the medical education community understand how to implement EPAs as a framework for competency demonstration, as educators try to determine what works, under what conditions and in what settings. Only through a committed effort to share lessons learned can the promise of the theory be translated to practice in the field.