Considerations that will determine if competency-based assessment is a sustainable innovation

Enabling Factors for the Implementation of Competency-Based Curricula in Colleges and Schools of Pharmacy

OBJECTIVES

To review the implementation drivers of competency-based pharmacy education (CBPE) and provide recommendations for enablers.

FINDINGS

Competency-based education is an emerging model in the health professions, focusing on time-variable competency development and achievement compared with a time-bound, course-based, traditional model. CBPE is an outcomes-based organized framework of competencies enabling pharmacists to meet health care and societal needs. However, challenges need to be recognized and overcome for the successful implementation of CBPE. Competency drivers include defining the competencies and roles of stakeholders, developing transparent learning trajectories and aligned assessments, and establishing lifetime development programs for stakeholders. Organization drivers include developing support systems for stakeholders; facilitating connections between all educational experiences; and having transparent assessment plans, policies, and procedures that align with core CBPE precepts, including the sustainability of time-variability. Leadership drivers include establishing growth mindset and facilitating a culture of connection between workplace and educational environments, program advocacy by institutional leaders, accepting failures as part of the process, shifting the organizational culture away from learner differentiation toward competence, and maintaining sufficient administrative capability to support CBPE.

SUMMARY

The successful implementation of CBPE involves enabling the competency, organization, and leadership drivers that will lead to program success. More research is needed in the areas of creation, implementation, and assessment of CBPE to determine success in this model. We have reviewed and provided recommendations to enable the drivers of successful implementation of CBPE.

Lessons learned spanning 17 years of experience with three consecutive nationwide competency based medical education training plans

Introduction

Curricula for postgraduate medical education have transformed since the introduction of competency based medical education (CBME). Postgraduate training plans offer broader training with different competencies and an outcome-based approach, in addition to the medical technical aspects of training. However, CBME also has its challenges. Over the past years, critical views have been shared on the potential drawbacks of CBME, such as assessment burden and conflicts with practicality in the workplace. Recent studies identified a need for a better understanding of how the evolving concept of CBME has been translated to curriculum design and implemented in the practice of postgraduate training. The aim of this study was to describe the development of CBME translations to curriculum design, based on three consecutive postgraduate training programs spanning 17 years.

Method

We performed a document analysis of three consecutive Dutch gynecology and obstetrics training plans that were implemented in 2005, 2013, and 2021. We used template analysis to identify changes over time.

Results

Over time, CBME-based curriculum design changed in several domains. Assessment changed from a model with a focus on summative decision to one with an emphasis on formative, low-stakes assessments aimed at supporting learning. The training plans evolved in parallel to evolving educational insights, e.g., by placing increasing emphasis on personal development. The curricula focused on a competency-based concept by introducing training modules and personalized authorization based on feedback rather than on a set duration of internships. There was increasing freedom in personalized training trajectories in the training plans, together with increasing trust towards the resident.

Conclusion

The way CBME was translated into training plans has evolved in the course of 17 years of experience with CMBE-based education. The main areas of change were the structure of the training plans, which became increasingly open, the degree to which learning outcomes were mandatory or not, and the way these outcomes were assessed.

Development, Implementation, and Meta-Evaluation of a National Approach to Programmatic Assessment in Canadian Family Medicine Residency Training

The growing international adoption of competency-based medical education has created a desire for descriptions of innovative assessment approaches that generate appropriate and sufficient information to allow for informed, defensible decisions about learner progress. In this article, the authors provide an overview of the development and implementation of the approach to programmatic assessment in postgraduate family medicine training programs in Canada, called Continuous Reflective Assessment for Training (CRAFT). CRAFT is a principles-guided, high-level approach to workplace-based assessment that was intentionally designed to be adaptable to local contexts, including size of program, resources available, and structural enablers and barriers. CRAFT has been implemented in all 17 Canadian family medicine residency programs, with each program taking advantage of the high-level nature of the CRAFT guidelines to create bespoke assessment processes and tools appropriate for their local contexts. Similarities and differences in CRAFT implementation between 5 different family medicine residency training programs, representing both English- and French-language programs from both Western and Eastern Canada, are described. Despite the intentional flexibility of the CRAFT guidelines, notable similarities in assessment processes and procedures across the 5 programs were seen. A meta-evaluation of findings from programs that have published evaluation information supports the value of CRAFT as an effective approach to programmatic assessment. While CRAFT is currently in place in family medicine residency programs in Canada, given its adaptability to different contexts as well as promising evaluation data, the CRAFT approach shows promise for application in other training environments.

The Importance of Professional Development in a Programmatic Assessment System: One Medical School’s Experience

The Cleveland Clinic Lerner College of Medicine of Case Western Reserve University (CCLCM) was created in 2004 as a 5-year undergraduate medical education program with a mission to produce future physician-investigators. CCLCM’s assessment system aligns with the principles of programmatic assessment. The curriculum is organized around nine competencies, where each competency has milestones that students use to self-assess their progress and performance. Throughout the program, students receive low-stakes feedback from a myriad of assessors across courses and contexts. With support of advisors, students construct portfolios to document their progress and performance. A separate promotion committee makes high-stakes promotion decisions after reviewing students’ portfolios. This case study describes a systematic approach to provide both student and faculty professional development essential for programmatic assessment. Facilitators, barriers, lessons learned, and future directions are discussed.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

On the validity of summative entrustment decisions

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Workplace-based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

OBJECTIVE

In the era of competency-based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace-based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees' WBAs in emergency medicine (EM).

METHODS

The authors conducted a scoping review in the style of Arksey and O'Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM.

RESULTS

Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone.

CONCLUSION

There is increasing interest in the areas around data collection and utilization in the age of CBME. The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.

The Value of Programmatic Assessment in Supporting Educators and Students to Succeed: A Qualitative Evaluation

BACKGROUND

Programmatic assessment has been proposed as the way forward for competency-based assessment, yet there is a dearth of literature describing the implementation and evaluation of programmatic assessment approaches.

OBJECTIVE

To evaluate the implementation of a programmatic assessment and explore its ability to support students and assessors.

DESIGN

A qualitative evaluation of programmatic assessment was employed.

PARTICIPANTS/SETTING

Interviews with graduates (n = 8) and preceptors (n = 12) together with focus groups with faculty assessors (n = 9) from the one Australian university explored experiences of the programmatic approach, role of assessment in learning, and defensibility of assessment decisions in determining competence.

ANALYSIS PERFORMED

Data were analyzed into key themes using framework analysis.

RESULTS

The programmatic assessment increased confidence in defensibility of assessment decisions, reduced emotional burden of assessment, increased value of assessment, and identified and remediated at-risk students earlier when philosophical and practice shifts in approaches to assessment were embraced.

CONCLUSIONS

Programmatic assessment supports a holistic approach to competency development and assessment and has multiple benefits for learners and assessors.

From prescription to guidance: a European framework for generic competencies

In postgraduate medical education, required competencies are described in detail in existing competency frameworks. This study proposes an alternative strategy for competency-based medical education design, which is supported by change management theories. We demonstrate the value of allowing room for re-invention and creative adaptation of innovations. This new strategy was explored for the development of a new generic competency framework for a harmonised European curriculum in Obstetrics and Gynaecology. The generic competency framework was developed through action research. Data were collected by four European stakeholder groups (patients, nurses, midwives and hospital boards), using a variety of methods. Subsequently, the data were analysed further in consensus discussions with European specialists and trainees in Obstetrics and Gynaecology. These discussions ensured that the framework provides guidance, is specialty-specific, and that implementation in all European countries could be feasible. The presented generic competency framework identifies four domains: 'Patient-centred care', 'Teamwork', 'System-based practice' and 'Personal and professional development'. For each of these four domains, guiding competencies were defined. The new generic competency framework is supported by European specialists and trainees in Obstetrics and Gynaecology, as well as by their European stakeholders. According to change management theories, it seems vital to allow room for re-invention and creative adaptation of the competency framework by medical professionals. Therefore, the generic competency framework offers guidance rather than prescription. The presented strategy for competency framework development offers leads for implementation of competency-based medical education as well as for development of innovations in postgraduate medical education in general.

Building a core competency assessment program for all stakeholders: the design and building of sailing ships can inform core competency frameworks

When educators are developing an effective and workable assessment program in graduate medical education by employing action research and stakeholder mapping to identify core competency domains and directives, the multi-stage process can be guided and informed by utilizing the story of designing, building and sea-testing sailing ships as a metaphor. However, the current challenge of physician burnout demands additional attention when formulating medical training frameworks, assessment guidelines and mentoring programs in 2020. The possibility of job-crafting is raised for consideration by designers of core competency frameworks in the health professions.

Entrustable Professional Activities in Psychiatry: A Systematic Review

OBJECTIVE

Entrustable professional activities (EPAs) represent discrete clinical tasks that can be entrusted to trainees in psychiatry. They are increasingly being used as educational framework in several countries. However, the empirical evidence available has not been synthesized in the field of psychiatry. Therefore, the authors conducted a systematic review in order to summarize and evaluate the available evidence in the field of EPAs in undergraduate and graduate medical education in psychiatry.

METHODS

The authors searched PubMed, Cochrane Library, ERIC, Embase, PsycINFO, all Ovid journals, Scopus, Web of Science, MedEdPORTAL, and the archives of Academic Psychiatry for articles reporting quantitative and qualitative research as well as educational case reports on EPAs in undergraduate and graduate psychiatry education published since 2005. All included articles were assessed for content (development, implementation, and assessment of EPAs) and quality using the Quality Assessment Tool for Studies with Diverse Designs.

RESULTS

The authors screened 2807 records and included a total of 20 articles in the final data extraction. Most studies were expert consensus reports (n = 6, 30%) and predominantly conducted in English-speaking countries (n = 17, 85%). Papers reported mainly EPA development and/or EPA implementation studies (n = 14, 70%), whereas EPA assessment studies were less frequent (n = 6, 30%). Publications per year showed an increasing trend both in quantity (from 1 in 2011 to 7 in 2018) and quality (from a QATSDD score of 27 in 2011 to an average score of 39 in 2018). The main focus of the articles was the development of individual EPAs for different levels of training for psychiatry or on curricular frameworks based on EPAs in psychiatry (n = 10, 50%). The lack of empirical controlled studies does currently not allow for meta-analyses of educational outcomes.

CONCLUSIONS

The concept of EPA-based curricula seems to become increasingly present, a focus in the specialty of psychiatry both in UME and GME. The lack of empirical research in this context is an important limitation for educational practice recommendations. Currently there is only preliminary but promising data available for using EPAs with regard to educational outcomes. EPAs seem to be effectively used from a curriculum design perspective for UME and GME in psychiatry.

Entrustment Ratings in Internal Medicine Training: Capturing Meaningful Supervision Decisions or Just Another Rating?

The implementation of Entrustable Professional Activities has led to the simultaneous development of assessment based on a supervisor's entrustment of a learner to perform these activities without supervision. While entrustment may be intuitive when we consider the direct observation of a procedural task, the current implementation of rating scales for internal medicine's non-procedural tasks, based on entrustability, may not translate into meaningful learner assessment. In these Perspectives, we outline a number of potential concerns with ad hoc entrustability assessments in internal medicine post-graduate training: differences in the scope of procedural vs. non-procedural tasks, acknowledgement of the type of clinical oversight common within internal medicine, and the limitations of entrustment language. We point towards potential directions for inquiry that would require us to clarify the purpose of the entrustability assessment, reconsider each of the fundamental concepts of entrustment in internal medicine supervision and explore the use of descriptive rather than numeric assessment approaches.