"The Most Crushing Thing": Understanding Resident Assessment Burden in a Competency-Based Curriculum

"I Think Many of Them Want to Appear to Have a Growth Mindset": Exploring Supervisors' Perceptions of Feedback-Seeking Behavior

PURPOSE

Feedback seeking is an expected learner competency. Motivations to seek feedback are well explored, but we know little about how supervisors perceive such requests for feedback. These perceptions matter because how supervisors judge requests can affect the feedback they give. This study explores how supervisors perceive and attribute motivations behind feedback requests to better understand the benefits and hazards of feedback seeking.

METHOD

Constructivist grounded theory was used to interview supervisors at the Temerty Faculty of Medicine, University of Toronto, from February 2020 to September 2022. Supervisors were asked to describe instances when they perceived feedback requests as being sincere or insincere, what led to their opinions, and how they responded. Transcripts were analyzed and coded in parallel with data collection; data analysis was guided by constant comparison.

RESULTS

Seventeen faculty were interviewed. Participants perceived 4 motivations when learners sought feedback: affirmation or praise; a desire to improve; an administrative requirement, such as getting forms filled out; and hidden purposes, such as making a good impression. These perceptions were based on assumptions regarding the framing of the initial request; timing; preexisting relationship with the learner; learner characteristics, such as insecurity; and learner reactions to feedback, particularly defensiveness. Although being asked for feedback was generally well received, some participants reported irritation at requests that were repetitive, were poorly timed, or did not appear sincere.

CONCLUSIONS

Feedback seeking may prompt supervisors to consider learners' motivations, potentially resulting in a set of entangled attributions, assumptions, and reactions that shape the feedback conversation in invisible and potentially problematic ways. Learners should consider these implications as they frame their feedback requests and be explicit about what they want and why they want it. Supervisors should monitor their responses, ask questions to clarify requests, and err on the side of assuming feedback-seeking requests are sincere.

Post competency training in standardized training of resident physicians and integrated postgraduates

We focus on the importance and necessity of post-competency-based training in medical education alongside its current challenges and opportunities. Integrating post competency training into standardized educational frameworks is increasingly recognized as a critical component of preparing residents and postgraduates for real-world clinical practice. Post competency training represents a commitment to excellence in medical education, striving to produce competent, skilled practitioners to meet the challenges of modern healthcare.

The Next Era of Assessment Within Medical Education: Exploring Intersections of Context and Implementation

In competency-based medical education (CBME), which is being embraced globally, the patient-learner-educator encounter occurs in a highly complex context which contributes to a wide range of assessment outcomes. Current and historical barriers to considering context in assessment include the existing post-positivist epistemological stance that values objectivity and validity evidence over the variability introduced by context. This is most evident in standardized testing. While always critical to medical education the impact of context on assessment is becoming more pronounced as many aspects of training diversify. This diversity includes an expanding interest beyond individual trainee competence to include the interdependency and collective nature of clinical competence and the growing awareness that medical education needs to be co-produced among a wider group of stakeholders. In this Eye Opener, we wish to consider: 1) How might we best account for the influence of context in the clinical competence assessment of individuals in medical education? and by doing so, 2) How could we usher in the next era of assessment that improves our ability to meet the dynamic needs of society and all its stakeholders? The purpose of this Eye Opener is thus two-fold. First, we conceptualize - from a variety of viewpoints, how we might address context in assessment of competence at the level of the individual learner. Second, we present recommendations that address how to approach implementation of a more contextualized competence assessment.

Competency-Based Cardiology Training: A Simple Approach to Improve Supervisor Completion of Entrustable Professional Activities

Background

Adult cardiology residency programs formally transitioned to Competency by Design (CBD) in July 2021. CBD was designed to establish clear learning expectations and increase opportunities for coaching; however, cited challenges include inconsistent participation by staff, and variable timelines for receiving feedback. This project was designed to implement a simple intervention to improve expiry rates and completion timelines of entrustable professional activities (EPAs).

Methods

EPAs triggered by cardiology residents at Dalhousie University between July 1, 2020 and February 28, 2023 were reviewed. The intervention consisted of performance reviews, including a grand rounds presentation, along with a personalized data set distributed to each staff supervisor, with individual statistics compared to group averages. Outcomes include EPAs completed per resident-months, time to completion, and percentage of expired EPAs.

Results

At 12 months postintervention, the percentage of expired EPAs decreased from 35.0% to 21.5% (odds ratio 0.51, CI 0.33-0.79; P = 0.03), and the time to completion decreased from 7.3 ± 5.99 days to 5.0 ± 5.78 days (difference -2.31, CI -3.55 to -1.07; P < 0.001). The number of EPAs completed per resident-months increased from 3.10 to 4.29 (rate difference 1.18; CI 0.64-1.72; P < 0.001), and the percentage of EPAs completed within the target time of 48 hours increased from 54.4% to 71.5% (OR 2.11, CI 1.27-3.50; P = 0.004).

Conclusions

Performance reviews in the form of a group presentation, along with the distribution of personalized data sets to supervisors, positively impacted EPA expiry rates, completion timelines, and completion rates, which helped facilitate the transition to CBD.

How Do Trainees Use EPAs to Regulate Their Learning in the Clinical Environment? A Grounded Theory Study

Introduction

Entrustable Professional Activities (EPAs) can potentially support self-regulated learning in the clinical environment. However, critics of EPAs express doubts as they see potential harms, like checkbox behaviour. This study explores how GP-trainees use EPAs in the clinical environment through the lens of self-regulated learning theory and addresses the question of whether EPAs help or hinder trainees' learning in a clinical environment.

Methods

Using constructivist grounded theory methodology, a purposive and theoretical sample of GP-trainees across different years of training were interviewed. Two PICTOR interviews were added to refine and confirm constructed theory. Data collection and analysis followed principles of constant comparative analysis.

Results and Discussion

Trainees experience both hindering and helping influences of EPAs and self-regulate their learning by balancing these influences throughout GP-placements. Three consecutive stages were constructed each with different use of EPAs: adaptation, taking control, and checking the boxes. EPAs were most helpful in the 'taking control' stage. EPAs hindered self-regulated learning most during the final stage of training as trainees had other learning goals and experienced assessment of EPAs as bureaucratic and demotivating. Regularly discussing EPAs with supervisors helped to focus on specific learning goals, create opportunities for learning, and generate task-oriented feedback.

Conclusion

EPAs can both help and hinder self-regulated learning. How trainees balance both influences changes over time. Therefore, placements need to be at least long enough to enable trainees to gain and maintain control of learning. Supervisors and teachers should assist trainees in balancing the hindering and helping influences of EPAs.

"The feedback dilemma"-provider and learner perspectives regarding the barriers and facilitators for giving and receiving feedback in medical education: A parallel mixed methods approach

BACKGROUND

Learning in medical education involves a multitude of practical tasks and skills that are amenable to feedback provision. Though passive feedback is given, there is a consistent gap in feedback provision and its receipt. This study aims to assess provider perspectives on feedback and learner attributes influencing the receipt of feedback in medical educational settings.

MATERIALS AND METHODS

A parallel mixed methods study was conducted in September 2023 at a tertiary care teaching institute. A convenience sample of 40 medical teachers comprising two faculties per department and 30 students were included. Focus group discussions (FGDs) were conducted with students from each academic year for assessing the student-level factors (facilitators and barriers) in the receipt of feedback.

RESULTS

Among the 40 medical teachers who were interviewed, the majority of 23 (57.5%) were assistant professors and nearly half of them (18; 45.0%) were below the age of 30 years. The majority of the respondents (28; 70.0%) were females, and 34 (85.0%) of them were postgraduates. Most of them (24; 60.0%) had worked for more than 10 years at the institute. It was observed that 80.0% of the respondents had given feedback to their students at some point in their careers. The major barriers for providing feedback were lack of curricular guidelines, fear of affective responses from students, burden of clinical, administrative work, and lack of perceived need by both students and teachers. DESTEP analysis of the student-level factors governing the receipt of feedback shows the effects of institutional ethics and culture, feedback model utilized, and the influence of learner behaviors, motivations, and teacher attributes.

CONCLUSION

The study elucidates mentor- and mentee-level influencers for providing and receiving feedback. Effective teacher-student partnerships along with an optimal skill set are required to recognize the need, opportunities, and processes to provide and seek feedback. Shifting the focus from feedback giving to active feedback seeking would be a step toward creating effective and pragmatic feedback systems.

Competency based medical education implementation at the institutional level: A cross-discipline comparative program evaluation

INTRODUCTION

As an early adopter of competency-based medical education (CBME) our postgraduate institution was uniquely positioned to analyze implementation experience data across programs, while keeping institutional factors constant. We described participants' experiences related to CBME implementation across programs derived from early program evaluation efforts within our setting.

METHODS

This evaluation focused on eight residency programs at a medium-sized academic institution in Canada. Participants (n = 175) included program leaders, faculty, and residents. The study consisted of 3 phases: (1) describing intended implementation; (2) documenting enacted implementation; and (3) comparing intended with enacted implementation to inform adaptations. Each program's findings were summarized in technical reports which were then analyzed thematically. Cross program data were organized by themes.

RESULTS

Six themes were identified. All groups emphasized the need for ongoing refinement of CBME resulting from shared tensions such as increased assessment burden. However, there were some disparate CBME-related experiences between programs such as the experience with entrustable professional activities, the interpretation of retrospective entrustment anchors, and quality of feedback.

CONCLUSION

We detected several cross-program successes and important challenges related to CBME. Our experience can inform other programs engaging in implementation and evaluation of CBME.

Standardizing clinician training and patient care in the neonatal neurocritical care: A step-by-step guide

Neonatal neurocritical care (NNCC) has emerged as an important specialty to address neurological conditions affecting newborns including a wide spectrum of brain injuries and developmental impairment. Despite the discipline's growth, variability in NNCC service delivery, patient care, and clinical training poses significant challenges and potentially adversely impacts patient outcomes. Variations in neuroprotective strategies, postnatal care, and training methodologies highlight the urgent need for a unified approach to optimize both short- and long-term neurodevelopmental outcomes for these vulnerable population. This paper presents strategic blueprints for establishing standardized NNCC clinical care and training programs focusing on collaborative effort across medical and allied health professions. By addressing these inconsistencies, the paper proposes that standardizing NNCC practices can significantly enhance the quality of care, streamline healthcare resource utilization, and improve neurodevelopmental outcome, thus paving the way for a new era of neonatal neurological care.

Is Competency-Based Medical Education being implemented as intended? Early lessons learned from Physical Medicine and Rehabilitation

Background

As competency-based medical education (CBME) curricula are introduced in residency programs across Canada, systematic evaluation efforts are needed to ensure fidelity of implementation. This study evaluated early outcomes of CBME implementation in one Canadian Physical Medicine and Rehabilitation program that was an early adopter of CBME, with an aim to inform continuous quality improvement initiatives and CBME implementation nationwide.

Methods

Using Rapid Evaluation methodology, informed by the CBME Core Components Framework, the intended outcomes of CBME were compared to actual outcomes.

Results

Results suggested that a culture of feedback and coaching already existed in this program prior to CBME implementation, yet faculty felt that CBME added a framework to support feedback. The small program size was valuable in fostering strong relationships and individualized learning. However, participants expressed concerns about CBME fostering a reductionist approach to the development of competence. Challenges existed with direct observation, clear expectations for off-service training experiences, and tracking trainee progress. There was trepidation surrounding national curricular change, yet the institution-wide approach to CBME implementation created shared experiences and a community of practice.

Conclusions

Program evaluation can help understand gaps between planned versus enacted implementation of CBME, and foster adaptations to improve the fidelity of implementation.

Chronicling the Transition to Competency-Based Medical Education in a Small Subspeciality Program

Background As medical education programs transition to competency-based medical education (CBME), experiences transitioning in the context of small subspecialty programs remain unknown, yet they are needed for effective implementation and continual improvements. Objective To examine faculty and resident experiences transitioning to CBME in a small subspeciality program. Methods Using a qualitative descriptive approach and constructivist lens, faculty and residents in McMaster University's geriatric psychiatry subspecialty program were interviewed about their transition experiences between November 2021 and February 2022, after the program's soft launch of CBME in 2020. Interviews were transcribed and data were analyzed using thematic analysis. Reflexive memo writing and investigator and data triangulation strategies were employed to ensure rigor and trustworthiness of the data. Results Ten of the 17 faculty members (59%) and 3 residents (100%) participated. Six themes were developed: (1) Both faculty and residents see themselves as somewhat knowledgeable about CBME, but sources of knowledge vary; (2) More frequent feedback is beneficial; (3) Aspects of CBME that are challenging for residents are beneficial for faculty; (4) Competence committees are perceived positively despite most participants' limited firsthand experience with them; (5) Small program size is both a barrier and facilitator to providing and receiving feedback; and (6) Suggestions for improvement are centered on helping manage faculty and resident workload imposed by CBME. Conclusions Incongruent expectations surrounding entrustable professional activity management were highlighted as an area requiring support. Collegial relationships among faculty and residents made it difficult for faculty to provide constructive feedback but improved residents' perceptions of the feedback.

Was it all worth it? A graduating resident perspective on CBME

BACKGROUND

Our institution simultaneously transitioned all postgraduate specialty training programs to competency-based medical education (CBME) curricula. We explored experiences of CBME-trained residents graduating from five-year programs to inform the continued evolution of CBME in Canada.

METHODS

We utilized qualitative description to explore residents' experiences and inform continued CBME improvement. Data were collected from fifteen residents from various specialties through focus groups, interviews, and written responses. The data were analyzed inductively, using conventional content analysis.

RESULTS

We identified five overarching themes. Three themes provided insight into residents' experiences with CBME, describing discrepancies between the intentions of CBME and how it was enacted, challenges with implementation, and variation in residents' experiences. Two themes - adaptations and recommendations - could inform meaningful refinements for CBME going forward.

CONCLUSIONS

Residents graduating from CBME training programs offered a balanced perspective, including criticism and recognition of the potential value of CBME when implemented as intended. Their experiences provide a better understanding of residents' needs within CBME curricula, including greater balance and flexibility within programs of assessment and curricula. Many challenges that residents faced with CBME could be alleviated by greater accountability at program, institutional, and national levels. We conclude with actionable recommendations for addressing residents' needs in CBME.

Motivations for Entrustable Professional Activity Assessment: Gaps Between Curriculum Theory and Resident Reality

Background Previous research demonstrates mixed reactions from residents toward competency-based medical education (CBME), and entrustable professional activities (EPAs) specifically. However, understanding what motivates residents to obtain EPAs may be vital to the longevity of CBME, given the emphasis on assessment for learning under this paradigm. Objective This study explored resident perspectives across 3 domains: motivation for obtaining EPAs, perceived importance of EPAs, and overall thoughts on CBME curriculum. Methods This was a sequential exploratory mixed-methods study involving 2 phases of data collection. Phase 1 was semi-structured interviews with residents enrolled in CBME at one Canadian institution from November 2019 to July 2020. Analyses included thematic and manifest content analysis. Phase 2 was an electronic close-ended survey to capture residents' primary motivation for requesting EPAs and importance of EPAs for learning. Survey data were analyzed descriptively. Results Of 120 eligible residents, 25 (21%) and 107 (89%) participated in the interview and survey, respectively. Program requirement was the dominant motivation for obtaining EPAs. There was variability in perceived importance of EPAs on learning. Increased resident workload, gaming the system to maximize EPA scores, and lack of shared ownership from preceptors were cited as critiques of the curriculum. Survey responses corroborated interview findings. Conclusions Although many residents recognize the value of EPAs, the majority are not intrinsically motivated to seek out assessment under the current CBME framework.

Competency-Based Time-Variable Anesthesiology Residency Training: Identification of Problems and Solutions

BACKGROUND

Global medical education is gradually moving toward more comprehensive implementations of a competency-based education (CBE) model. Elimination of standard time-based training and adoption of time-variable training (competency-based time-variable training [CB-TVT]) is one of the final stages of implementation of CBE. While CB-TVT has been implemented in some programs outside the United States, residency programs in the United States are still exploring this approach to training. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) are encouraging member boards and residency review committees to consider innovative ways programs could implement CB-TVT. The goals of this study were to (1) identify potential problems with the implementation of CB-TVT in anesthesiology residency training, (2) rank the importance of the problems and the perceived difficulty of solving them, and (3) develop proposed solutions to the identified problems.

METHODS

Study participants were recruited from key stakeholder groups in anesthesiology education, including current or former program directors, department chairs, residents, fellows, American Board of Anesthesiology (ABA) board members, ACGME residency review committee members or ACGME leaders, designated institutional officials, residency program coordinators, clinical operations directors, and leaders of large anesthesiology community practice groups. This study was conducted in 2 phases. In phase 1, survey questionnaires were iteratively distributed to participants to identify problems with the implementation of CB-TVT. Participants were also asked to rank the perceived importance and difficulty of each problem and to identify relevant stakeholder groups that would be responsible for solving each problem. In phase 2, surveys focused on identifying potential solutions for problems identified in phase 1.

RESULTS

A total of 36 stakeholders identified 39 potential problems, grouped into 7 major categories, with the implementation of CB-TVT in anesthesiology residency training. Of the 39 problems, 19 (48.7%) were marked as important or very important on a 5-point scale and 12 of 19 (63.2%) of the important problems were marked as difficult or very difficult to solve on a 5-point scale. Stakeholders proposed 165 total solutions to the identified problems.

CONCLUSIONS

CB-TVT is a promising educational model for anesthesiology residency, which potentially results in learner flexibility, individualization of curricula, and utilization of competencies to determine learner advancement. Because of the potential problems with the implementation of CB-TVT, it is important for future pilot implementations of CB-TVT to document realized problems, efficacy of solutions, and effects on educational outcomes to justify the burden of implementing CB-TVT.

Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

"Doing well": Intraoperative entrustable professional activity assessments provided limited technical feedback

Background

Entrustable Professional Activities (EPAs) allow for the assessment of specific, observable, essential tasks in medical education. Since being developed in non-surgical fields, EPA assessments have been implemented in surgery to explore intraoperative entrustment. However, assessment burden is a significant problem for faculty, and it is unknown whether EPA assessments enable formative technical feedback. EPAs' formative utility could inform how surgical programs facilitate technical feedback for trainees. We aimed to assess the extent to which narrative comments provided through the Fellowship Council (FC) EPA assessments contained technical feedback.

Methods

The FC previously collected EPA assessments for subspecialty surgical fellows from September 2020 to October 2022. Two raters reviewed assessments' narrative comments for inclusion of each skill area that makes up part of the Objective Structured Assessment of Technical Skills (OSATS). A third rater reconciled discrepant ratings.

Results

During the study period, there were 3302 completed EPA assessments, including 1191 fellow self-assessments, 1124 faculty assessments, and 987 assessments without an identified assessor role. We found that assessments' narrative comments related to a median of two of the seven OSATS areas (IQR:1-2). There were no comments relevant to any of the seven OSATS areas in 16.0 % of all assessments.

Conclusions

In this review of narrative comments for EPA assessments from the FC, we found that limited technical feedback of the kind included in the OSATS was provided in many assessments. These results suggest benefit to adjusting the EPA form, enhancing faculty development, or continuing additional types of targeted technical assessment intraoperatively.

Key message

This analysis of narrative comments from fellowship EPA assessments showed that many assessments included limited technical feedback. To allow for continued technical feedback for fellows, these results highlight the need for further refinements of the EPA assessment form, additional faculty development, or ongoing use of other types of technical assessment.

Evaluating Competence by Design as a Large System Change Initiative: Readiness, Fidelity, and Outcomes

Program evaluation is an essential, but often neglected, activity in any transformational educational change. Competence by Design was a large-scale change initiative to implement a competency-based time-variable educational system in Canadian postgraduate medical education. A program evaluation strategy was an integral part of the build and implementation plan for CBD from the beginning, providing insights into implementation progress, challenges, unexpected outcomes, and impact. The Competence by Design program evaluation strategy was built upon a logic model and three pillars of evaluation: readiness to implement, fidelity and integrity of implementation, and outcomes of implementation. The program evaluation strategy harvested from both internally driven studies and those performed by partners and invested others. A dashboard for the program evaluation strategy was created to transparently display a real-time view of Competence by Design implementation and facilitate continuous adaptation and improvement. The findings of the program evaluation for Competence by Design drove changes to all aspects of the Competence by Design implementation, aided engagement of partners, supported change management, and deepened our understanding of the journey required for transformational educational change in a complex national postgraduate medical education system. The program evaluation strategy for Competence by Design provides a framework for program evaluation for any large-scale change in health professions education.

Implementing Competence Committees on a National Scale: Design and Lessons Learned

Competence committees (CCs) are a recent innovation to improve assessment decision-making in health professions education. CCs enable a group of trained, dedicated educators to review a portfolio of observations about a learner's progress toward competence and make systematic assessment decisions. CCs are aligned with competency based medical education (CBME) and programmatic assessment. While there is an emerging literature on CCs, little has been published on their system-wide implementation. National-scale implementation of CCs is complex, owing to the culture change that underlies this shift in assessment paradigm and the logistics and skills needed to enable it. We present the Royal College of Physicians and Surgeons of Canada's experience implementing a national CC model, the challenges the Royal College faced, and some strategies to address them. With large scale CC implementation, managing the tension between standardization and flexibility is a fundamental issue that needs to be anticipated and addressed, with careful consideration of individual program needs, resources, and engagement of invested groups. If implementation is to take place in a wide variety of contexts, an approach that uses multiple engagement and communication strategies to allow for local adaptations is needed. Large-scale implementation of CCs, like any transformative initiative, does not occur at a single point but is an evolutionary process requiring both upfront resources and ongoing support. As such, it is important to consider embedding a plan for program evaluation at the outset. We hope these shared lessons will be of value to other educators who are considering a large-scale CBME CC implementation.

Exploring the Quality of Feedback in Entrustable Professional Activity Narratives Across 24 Residency Training Programs

Background Competency-based medical education (CBME) has been implemented in many residency training programs across Canada. A key component of CBME is documentation of frequent low-stakes workplace-based assessments to track trainee progression over time. Critically, the quality of narrative feedback is imperative for trainees to accumulate a body of evidence of their progress. Suboptimal narrative feedback will challenge accurate decision-making, such as promotion to the next stage of training. Objective To explore the quality of documented feedback provided on workplace-based assessments by examining and scoring narrative comments using a published quality scoring framework. Methods We employed a retrospective cohort secondary analysis of existing data using a sample of 25% of entrustable professional activity (EPA) observations from trainee portfolios from 24 programs in one institution in Canada from July 2019 to June 2020. Statistical analyses explore the variance of scores between programs (Kruskal-Wallis rank sum test) and potential associations between program size, CBME launch year, and medical versus surgical specialties (Spearman's rho). Results Mean quality scores of 5681 narrative comments ranged from 2.0±1.2 to 3.4±1.4 out of 5 across programs. A significant and moderate difference in the quality of feedback across programs was identified (χ2=321.38, P<.001, ε2=0.06). Smaller programs and those with an earlier launch year performed better (P<.001). No significant difference was found in quality score when comparing surgical/procedural and medical programs that transitioned to CBME in this institution (P=.65). Conclusions This study illustrates the complexity of examining the quality of narrative comments provided to trainees through EPA assessments.

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Competency-based medical education: The spark to ignite healthcare's escape fire

High-value care is what patients deserve and what healthcare professionals should deliver. However, it is not what happens much of the time. Quality improvement master Dr. Don Berwick argued more than two decades ago that American healthcare needs an escape fire, which is a new way of seeing and acting in a crisis situation. While coined in the U.S. context, the analogy applies in other Western healthcare contexts as well. Therefore, in this paper, the authors revisit Berwick's analogy, arguing that medical education can, and should, provide the spark for such an escape fire across the globe. They assert that medical education can achieve this by fully embracing competency-based medical education (CBME) as a way to place medicine's focus on the patient. CBME targets training outcomes that prepare graduates to optimize patient care. The authors use the escape fire analogy to argue that medical educators must drop long-held approaches and tools; treat CBME implementation as an adaptive challenge rather than a technical fix; demand genuine, rich discussions and engagement about the path forward; and, above all, center the patient in all they do.

Only as Strong as the Weakest Link: Resident Perspectives on Entrustable Professional Activities and Their Impact on Learning

Background Core to competency-based medical education (CBME) is the use of frequent low-stakes workplace-based assessments. In the Canadian context, these observations of performance are framed around entrustable professional activities (EPAs). Objective We aimed to explore residents' real-world perspectives of EPAs and their perceived impact on learning, because assessments perceived to be "inauthentic," or not truly reflective of their lived experiences, may interfere with learning. Methods Using constructivist grounded theory, we conducted 18 semistructured interviews in 2021 with residents from all programs that had implemented CBME at one tertiary care academic center in Canada. Participants were recruited via email through respective program administrators. Data collection and analysis occurred iteratively, and categories were identified using constant comparative analysis. Results Residents were strikingly polarized, perceiving EPAs as either a valuable opportunity for professional growth or as an onerous requirement that interfered with learning. Regardless of what view participants held, all perspectives were informed by: (1) the program administration and the perceived messaging from program to residents; (2) faculty assessors and their perceived degree of engagement, or "buy-in" with the EPA system; and ultimately (3) learner behavior. We theorized from these findings that all 3 aspects must be working in tandem for the assessment system to function as intended. Conclusions From the learners' perspective, there exists a dynamic, interdependent relationship between the 3 CBME stakeholders. As such, the perceived value of the EPA assessment system can only be as strong as the weakest link in the chain.

"Rater training" re-imagined for work-based assessment in medical education

In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

OBJECTIVE

Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment.

DESIGN

The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures.

SETTING

This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative.

PARTICIPANTS

A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded.

RESULTS

The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures.

CONCLUSIONS

Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.

Assessing operative competence in core surgical training: A reliability analysis

BACKGROUND

This study quantifies the number of observations required to reliably assess the operative competence of Core Surgical Trainees (CSTs) in Ireland, using the Supervised Structured Assessment of Operative Performance (SSAOP) tool.

METHODS

SSAOPs (April 2016-February 2021) were analysed across a mix of undifferentiated procedures, as well as for three commonly performed general surgery procedures in CST: appendicectomy, abdominal wall hernia repair, and skin/subcutaneous lesion excision. Generalizability and Decision studies determined the number of observations required to achieve dependability indices ≥0.8, appropriate for use in high-stakes assessment.

RESULTS

A total of 2,294 SSAOPs were analysed. Four assessors, each observing 10 cases, can generate scores sufficiently reliable for use in high-stakes assessments. Focusing on a selection of core procedures yields more favourable reliability indices.

CONCLUSION

Trainers should conduct repeated assessments across a smaller number of procedures to improve reliability. Programs should increase the assessor mix to yield sufficient dependability indices for high-stakes assessment.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

The Assessment Burden in Competency-Based Medical Education: How Programs Are Adapting

Residents and faculty have described a burden of assessment related to the implementation of competency-based medical education (CBME), which may undermine its benefits. Although this concerning signal has been identified, little has been done to identify adaptations to address this problem. Grounded in an analysis of an early Canadian pan-institutional CBME adopter's experience, this article describes postgraduate programs' adaptations related to the challenges of assessment in CBME. From June 2019-September 2022, 8 residency programs underwent a standardized Rapid Evaluation guided by the Core Components Framework (CCF). Sixty interviews and 18 focus groups were held with invested partners. Transcripts were analyzed abductively using CCF, and ideal implementation was compared with enacted implementation. These findings were then shared back with program leaders, adaptations were subsequently developed, and technical reports were generated for each program. Researchers reviewed the technical reports to identify themes related to the burden of assessment with a subsequent focus on identifying adaptations across programs. Three themes were identified: (1) disparate mental models of assessment processes in CBME, (2) challenges in workplace-based assessment processes, and (3) challenges in performance review and decision making. Theme 1 included entrustment interpretation and lack of shared mindset for performance standards. Adaptations included revising entrustment scales, faculty development, and formalizing resident membership. Theme 2 involved direct observation, timeliness of assessment completion, and feedback quality. Adaptations included alternative assessment strategies beyond entrustable professional activity forms and proactive assessment planning. Theme 3 related to resident data monitoring and competence committee decision making. Adaptations included adding resident representatives to the competence committee and assessment platform enhancements. These adaptations represent responses to the concerning signal of significant burden of assessment within CBME being experienced broadly. The authors hope other programs may learn from their institution's experience and navigate the CBME-related assessment burden their invested partners may be facing.

Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures

BACKGROUND

Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric.

METHODS

The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28 years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2.

RESULTS

The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively).

CONCLUSION

The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.

Emergency medicine resident perceptions of simulation-based training and assessment in competence by design

OBJECTIVES

With the launch of competence by design (CBD) in emergency medicine (EM) in Canada, there are growing recommendations on the use of simulation for the training and assessment of residents. Many of these recommendations have been suggested by educational leaders and often exclude the resident stakeholder. This study sought to explore their experiences and perceptions of simulation in CBD.

METHODS

Qualitative data were collected from November 2020 to May 2021 at McMaster University and the University of Toronto after receiving ethics approval from both sites. Eligible participants included EM residents who were interviewed by a trained interviewer using a semi-structured interview guide. All interviews were recorded, transcribed, coded, and collapsed into themes. Data analysis was guided by constructivist grounded theory.

RESULTS

A total of seventeen residents participated. Thematic analysis revealed three major themes: 1) impact of CBD on resident views of simulation; 2) simulation's role in obtaining entrustable professional activities (EPAs) and filling educational gaps; and 3) conflicting feelings on the use of high-stakes simulation-based assessment in CBD.

CONCLUSIONS

EM residents strongly support using simulation in CBD and acknowledge its ability to bridge educational gaps and fulfill specific EPAs. However, this study suggests some unintended consequences of CBD and conflicting views around simulation-based assessment that challenge resident perceptions of simulation as a safe learning space. As CBD evolves, educational leaders should consider these impacts when making future curricular changes or recommendations.

Clinical Assessment of Residents: A Survey of Clinician Educators Regarding Resident Assessment Burden and Modifiable Factors

BACKGROUND

A fundamental role of the clinician educator is to provide thoughtful assessments for resident development. A gap in the literature exists about whether the completion of assessments contributes to clinician educator burden.

OBJECTIVE

We sought to understand the degree to which completing resident assessments contributes to clinician educator burden, the drivers behind such perception, and whether modifiable factors exist.

METHODS

In October 2020, we conducted a cross-sectional study of adult hospital medicine clinician educators to explore burden associated with resident assessment. The authors developed a 10-item electronic survey (Likert type and sliding scale responses), asking about demographics, context, frequency and degree of burden, burdensome aspects of assessments, estimated time for assessments, and percentage of assessments turned in late or never. We conducted subgroup analyses for differences in responses based on sex and number of years practicing, and regression analyses for predictors of burden degree.

RESULTS

Fifty of 81 (62%) surveyed faculty responded. Two percent (1 of 50) reported no burden, while 42% (21 of 50) reported infrequent ("never," "rarely," "sometimes") and 56% (28 of 50) reported frequent ("often," "always") burden. Of those experiencing burden, 67% (33 of 49) reported slight or moderate, and 33% (16 of 49) reported significant or extreme burden. Potentially modifiable causes included assessment request boluses, lag time between resident service and assessment requests, and technology involved. Female clinician educators estimated submitting a higher percentage of late assessments than males (65% vs 41%, P=.02). Number of years practicing was inversely associated with assessment time (β=-0.28, P=.01).

CONCLUSIONS

Our findings suggest that resident assessments are a source of burden among adult hospital medicine clinician educators and that several potentially modifiable factors may underlie this burden.

Exploring the Quality of Narrative Feedback Provided to Residents During Ambulatory Patient Care in Medicine and Surgery

OBJECTIVES

The transition to competency-based medical education (CBME) has increased the volume of residents' assessment data; however, the quality of the narrative feedback is yet to be used as feedback-on-feedback for faculty. Our objectives were (1) to explore and compare the quality and content of narrative feedback provided to residents in medicine and surgery during ambulatory patient care and (2) to use the Deliberately Developmental Organization framework to identify strengths, weaknesses, and opportunities to improve quality of feedback within CBME.

METHODS

We conducted a mixed convergent methods study with residents from the Departments of Surgery (DoS; n = 7) and Medicine (DoM; n = 9) at Queen's University. We used thematic analysis and the Quality of Assessment for Learning (QuAL) tool to analyze the content and quality of narrative feedback documented in entrustable professional activities (EPAs) assessments for ambulatory care. We also examined the association between the basis of assessment, time to provide feedback, and the quality of narrative feedback.

RESULTS

Forty-one EPA assessments were included in the analysis. Three major themes arose from thematic analysis: Communication, Diagnostics/Management, and Next Steps. Quality of the narrative feedback varied; 46% had sufficient evidence about residents' performance; 39% provided a suggestion for improvement; and 11% provided a connection between the suggestion and the evidence. There were significant differences between DoM and DoS in quality of feedback scores for evidence (2.1 [1.3] vs. 1.3 [1.1]; p < 0.01) and connection (0.4 [0.5] vs. 0.1 [0.3]; p  =  0.04) domains of the QuAL tool. Feedback quality was not associated with the basis of assessment or time taken to provide feedback.

CONCLUSION

The quality of the narrative feedback provided to residents during ambulatory patient care was variable with the greatest gap in providing connections between suggestions and evidence about residents' performance. There is a need for ongoing faculty development to improve the quality of narrative feedback provided to residents.