It's a 'two-way street': resident perspectives of effective coaching relationships in the clinical learning environment

BACKGROUND

Coaching has gained traction in postgraduate medical education to enhance interactions between residents and clinical teachers, but these relationships present unique challenges and tensions. In order to realize the promises of coaching in medical education, we must understand how coaching relationships can be enacted to optimize resident development. The purpose of this study was to explore residents' perceptions of key characteristics for effective clinical teacher-resident (CT-R) coaching relationships.

METHODS

We conducted four focus groups and eight interviews with residents at a Canadian academic center. Using a social constructionist approach, focus groups and interviews were audio recorded, transcribed verbatim, and thematically analyzed.

RESULTS

Residents described three main characteristics that contributed to effective CT-R coaching relationships: safe, meaningful, and collaborative. Residents emphasized that these characteristics needed to be bidirectional in nature to be most effective, in that both the resident and clinical teacher embodied these characteristics.

CONCLUSIONS

Residents identified that effective coaching relationships were shaped not only by clinical teacher behaviours, but importantly, the quality of the interpersonal relationship that was fostered. Thus, it is imperative to consider the bidirectional nature of the CT-R coaching relationship when striving to enhance resident development.

Operationalizing Programmatic Assessment: The CBME Programmatic Assessment Practice Guidelines

PROBLEM

Assessing the development and achievement of competence requires multiple formative and summative assessment strategies and the coordinated efforts of trainees and faculty (who often serve in multiple roles, such as academic advisors, program directors, and competency committee members). Operationalizing programmatic assessment (PA) in competency-based medical education (CBME) requires comprehensive practice guidelines, written in accessible language with descriptions of stakeholder activities, to move assessment theory into practice and to help guide the trainees and faculty who enact PA.

APPROACH

Informed by the Appraisal of Guidelines for Research and Evaluation II (AGREE II) framework, the authors used a multiphase, multimethod approach to develop the CBME Programmatic Assessment Practice Guidelines (PA Guidelines). The 9 guidelines are organized by phases of assessment and include descriptions of stakeholder activities. A user guide provides a glossary of key terms and summarizes how the guidelines can be used by different stakeholder groups across postgraduate medical education (PGME) contexts. The 4 phases of guideline development, including internal stakeholder consultations and external expert review, occurred between August 2016 and March 2020.

OUTCOMES

Local stakeholders and external experts agreed that the PA Guidelines hold potential for guiding initial operationalization and ongoing refinement of PA in CBME by individual stakeholders, residency programs, and PGME institutions. Since July 2020, the PA Guidelines have been used at Queen's University to inform faculty and resident development initiatives, including online CBME modules for faculty, workshops for academic advisors/competence committee members, and a guide that supports incoming residents' transition to CBME.

NEXT STEPS

Research exploring the use of the PA Guidelines and user guide in multiple programs and institutions will gather further evidence of their acceptability and utility for guiding operationalization of PA in different contexts.

Seven ways to get a grip on implementing Competency-Based Medical Education at the program level

Competency-based medical education (CBME) curricula are becoming increasingly common in graduate medical education. Put simply, CBME is focused on educational outcomes, is independent of methods and time, and is composed of achievable competencies.¹ In spite of widespread uptake, there remains much to learn about implementing CBME at the program level. Leveraging the collective experience of program leaders at Queen's University, where CBME simultaneously launched across 29 specialty programs in 2017, this paper leverages change management theory to provide a short summary of how program leaders can navigate the successful preparation, launch, and initial implementation of CBME within their residency programs.

Competency-based education calls for programmatic assessment: But what does this look like in practice?

RATIONALE, AIMS, AND OBJECTIVES

Programmatic assessment has been identified as a system-oriented approach to achieving the multiple purposes for assessment within Competency-Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well-established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the 'two communities' metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization.

METHOD

An interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency-based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi-structured group and individual interviews conducted at nine months post-CBME implementation. Data were analyzed using a combination of data-based inductive analysis and theory-derived deductive analysis.

RESULTS

In this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low-stakes formative assessments to inform high-stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops.

CONCLUSIONS

The findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.

Reclaiming physician identity: It's time to integrate 'Doctor as Person' into the CanMEDS framework

In 1996, the Royal College of Physicians & Surgeons of Canada (RCPSC) adopted the CanMEDS framework with seven key roles: medical expert, communicator, collaborator, health advocate, manager, professional, and scholar. For many years, CanMEDS has been recognized around the world for defining what patients need from their physicians. From the start, the RCPSC acknowledged that these roles should  evolve over time to continue to meet patient and societal needs (updates in 2005 & 2015).  We propose that  an 8th role is now needed in the framework: “Doctor as Person”. Interestingly, this role was present in the foundational work through the Educating Future Physicians for Ontario (EFPO) project that the RCPSC drew upon in creating CanMEDS more than 20 years ago. Given today’s challenges of providing care in an increasingly stressed Canadian healthcare system, physicians are struggling more than ever with health and wellness, burnout, and the deterioration of the clinical environment. From the patient perspective, there is growing concern that physician-patient interactions are becoming increasingly impersonal and decreasingly patient-centered. The crack emerging in the foundation of physician identity needs to be remedied. We need to pay close attention to how we define ourselves as physicians, by better identifying the competencies required to navigate the personal and professional challenges we face. Only in so doing can we ward off the threat that exists in losing authentic human to human care interactions. Formalizing Doctor as Person as an 8th role in the CanMEDS framework will help patients and physicians create the space to have essential conversations about the humanity of medical care. 

Training disrupted: Practical tips for supporting competency-based medical education during the COVID-19 pandemic

The COVID-19 pandemic has disrupted healthcare systems around the world, impacting how we deliver medical education. The normal day-to-day routines have been altered for a number of reasons, including changes to scheduled training rotations, physical distancing requirements, trainee redeployment, and heightened level of concern. Medical educators will likely need to adapt their programs to maximize learning, maintain effective care delivery, and ensure competent graduates. Along with a continued focus on learner/faculty wellness, medical educators will have to optimize existing training experiences, adapt those that are no longer viable, employ new technologies, and be flexible when assessing competencies. These practical tips offer guidance on how to adapt medical education programs within the constraints of the pandemic landscape, stressing the need for communication, innovation, collaboration, flexibility, and planning within the era of competency-based medical education.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation

PURPOSE

Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change.

METHOD

Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences.

RESULTS

In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow.

CONCLUSIONS

Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Ten ways to get a grip on resident co-production within medical education change

The Royal College of Physicians and Surgeons of Canada (RCPSC) is transforming its national approach to postgraduate medical education by transitioning all specialty programs to competency based medical education (CBME) curriculums over a seven-year period. Queen's University, with special permission from the RCPSC, launched CBME curricula for all incoming residents across its 29 specialty programs in July 2017. Resident engagement, empowerment, and co-production through this transition has been instrumental in successful implementation of CBME at Queen's University. This article aims to use our own experience at Queen's in the context of current literature and rooted in change leadership theory, to provide a guide for educators, learners, and institutions on how to leverage the interest and enthusiasm of trainees in the transition to CBME in postgraduate training. The following ten tips provides a model for avoiding the "black ice" type pitfalls that can arise with learner involvement and ensure a smoother transition for other institutions moving forward with CBME implementation.

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring

Background

Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED.

Methods

The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen's simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson's correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback.

Results

There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills.

Conclusions

In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.

Impact of night shifts on emergency medicine resident resuscitation performance

AIM

Emergency medicine (EM) trainees often work nightshifts. We sought to measure how this circadian disruption affects EM resident performance during simulated resuscitations.

METHODS

This retrospective cohort study enrolled EM residents at a single Canadian academic centre over a six-year period. Residents completed twice-annual simulation-based resuscitation-focused objective structured clinical examinations (OSCEs) with assessment in four domains (primary assessment, diagnostic actions, therapeutic actions and communication), and a global assessment score (GAS). Primary and secondary exposures of interest were the presence of a nightshift (late-evening shifts ending between midnight and 03h00 or overnight shifts ending after 06h00) the day before or within three days before an OSCE. A random effects linear regression model was used to quantify the association between nightshifts and OSCE scores.

RESULTS

From 57 residents, 136 OSCE scores were collected. Working a nightshift the day before an OSCE did not affect male trainee scores but was associated with a significant absolute decrease in mean total scores (-6% [95% CI -12% to 0%]), GAS (-7% [-13% to 0%]), and communication (-9% [-16% to -2%]) scores among women. Working any nightshift within three days before an OSCE lowered absolute mean total scores by 4% [-7% to 0%] and communication scores by 5% [-5% to 0%] irrespective of gender.

CONCLUSION

Our results suggest that shift work may impact EM resident resuscitation performance, particularly in the communication domain. This impact may be more significant in women than men, suggesting a need for further investigation.

Comparison of Simulation-based Resuscitation Performance Assessments With In-training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study

OBJECTIVE

Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in greater detail the inferences that may be drawn from a simulation-based OSCE assessment.

METHODS

A prospective observational study was employed to explore the use of a multicenter simulation-based OSCE for evaluation of resuscitation competence. EM residents from five Canadian academic sites participated in the OSCE. Video-recorded performances were scored by blinded raters using the scenario-specific QSATs with domain-specific anchored scores (primary assessment, diagnostic actions, therapeutic actions, communication) and a global assessment score (GAS). Residents' portfolios were evaluated using a modified ITER subdivided by CanMEDS roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) and a GAS. Correlational and regression analyses were performed comparing components of each of the assessment methods.

RESULTS

Portfolio review and ITER scoring was performed for 79 residents participating in the simulation-based OSCE. There was a significant positive correlation between total OSCE and ITER scores (r = 0.341). The strongest correlations were found between ITER medical expert score and each of the OSCE GAS (r = 0.420), communication (r = 0.443), and therapeutic action (r = 0.484) domains. ITER medical expert was a significant predictor of OSCE total (p = 0.002). OSCE therapeutic action was a significant predictor of ITER total (p = 0.02).

CONCLUSIONS

Simulation-based resuscitation OSCEs and portfolio assessment captured by ITERs appear to measure differing aspects of competence, with weak to moderate correlation between those measures of conceptually similar constructs. In a program of competency-based assessment of EM residents, a simulation-based OSCE using the QSAT shows promise as a tool for assessing medical expert and communicator roles.

Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees - a Canadian multi-centred study

BACKGROUND

The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees.

METHOD

EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE. Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen's Simulation Assessment Tool (QSAT). Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario. A fully crossed generalizability study was also conducted for each examination centre.

RESULTS

Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres.

CONCLUSIONS

This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

The Simulation Olympics: a resuscitation-based simulation competition as an educational intervention

The Department of Emergency Medicine at Queen's University developed, implemented, and evaluated an interprofessional simulation-based competition called the Simulation Olympics with the purpose of encouraging health care providers to practice resuscitation skills and foster strong team-based attitudes. Eleven teams (N  =  45) participated in the competition. Teams completed three standardized resuscitation scenarios in a high-fidelity simulation laboratory with teams composed of nurses, respiratory therapists, and undergraduate and postgraduate medical trainees. Trained standardized actors and a dedicated technician were used for all scenarios. Judges evaluated team performance using standardized assessment tools. All participants (100%) completed an anonymous two-page questionnaire prior to the competition assessing baseline characteristics and evaluating participant attitudes, motivation, and barriers to participation. The majority of participants (71%) completed an evaluation form following the event focusing on highlights, barriers to participation, and desired future directions. Evaluations were uniformly positive in short-answer feedback and attitudinal scoring measures. To our knowledge, the Simulation Olympics competition is the first of its kind in Canada to be offered at an academic teaching hospital.