Exploring resident perceptions of initial competency based medical education implementation

Was it all worth it? A graduating resident perspective on CBME

BACKGROUND

Our institution simultaneously transitioned all postgraduate specialty training programs to competency-based medical education (CBME) curricula. We explored experiences of CBME-trained residents graduating from five-year programs to inform the continued evolution of CBME in Canada.

METHODS

We utilized qualitative description to explore residents' experiences and inform continued CBME improvement. Data were collected from fifteen residents from various specialties through focus groups, interviews, and written responses. The data were analyzed inductively, using conventional content analysis.

RESULTS

We identified five overarching themes. Three themes provided insight into residents' experiences with CBME, describing discrepancies between the intentions of CBME and how it was enacted, challenges with implementation, and variation in residents' experiences. Two themes - adaptations and recommendations - could inform meaningful refinements for CBME going forward.

CONCLUSIONS

Residents graduating from CBME training programs offered a balanced perspective, including criticism and recognition of the potential value of CBME when implemented as intended. Their experiences provide a better understanding of residents' needs within CBME curricula, including greater balance and flexibility within programs of assessment and curricula. Many challenges that residents faced with CBME could be alleviated by greater accountability at program, institutional, and national levels. We conclude with actionable recommendations for addressing residents' needs in CBME.

Strategies to Enable Transformation in Medical Education: Faculty and Trainee Development in Competence By Design

Transformative changes in health professions education need to incorporate effective faculty development, but few very large-scale faculty development designs have been described. The Royal College of Physicians and Surgeons of Canada's Competence by Design project was launched to transform the delivery of postgraduate medical education in Canada using a competency-based model. In this paper we outline the goals, principles, and rationale of the Royal College's national strategy for faculty and resident development initiatives to support the implementation of Competence by Design. We describe the activities and resources for both faculty and trainees that facilitated the redesign of training programs for each specialty and subspecialty at the national level, as well as supporting the implementation of the redesign at the local level. This undertaking was not without its challenges: we thus reflect on those challenges, enablers, and the lessons learned, and discuss a continuous quality improvement approach that was taken to iteratively inform the implementation process moving forward.

Exploring the Quality of Feedback in Entrustable Professional Activity Narratives Across 24 Residency Training Programs

Background Competency-based medical education (CBME) has been implemented in many residency training programs across Canada. A key component of CBME is documentation of frequent low-stakes workplace-based assessments to track trainee progression over time. Critically, the quality of narrative feedback is imperative for trainees to accumulate a body of evidence of their progress. Suboptimal narrative feedback will challenge accurate decision-making, such as promotion to the next stage of training. Objective To explore the quality of documented feedback provided on workplace-based assessments by examining and scoring narrative comments using a published quality scoring framework. Methods We employed a retrospective cohort secondary analysis of existing data using a sample of 25% of entrustable professional activity (EPA) observations from trainee portfolios from 24 programs in one institution in Canada from July 2019 to June 2020. Statistical analyses explore the variance of scores between programs (Kruskal-Wallis rank sum test) and potential associations between program size, CBME launch year, and medical versus surgical specialties (Spearman's rho). Results Mean quality scores of 5681 narrative comments ranged from 2.0±1.2 to 3.4±1.4 out of 5 across programs. A significant and moderate difference in the quality of feedback across programs was identified (χ2=321.38, P<.001, ε2=0.06). Smaller programs and those with an earlier launch year performed better (P<.001). No significant difference was found in quality score when comparing surgical/procedural and medical programs that transitioned to CBME in this institution (P=.65). Conclusions This study illustrates the complexity of examining the quality of narrative comments provided to trainees through EPA assessments.

The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study

Background

Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs.

Methods

Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall's tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients.

Results

Both the faculty's utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees' utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88).

Conclusions

The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.

Exploring stakeholder perspectives regarding the implementation of competency-based medical education: a qualitative descriptive study

Introduction

Competency-based medical education (CBME) offers perceived advantages and benefits for postgraduate medical education (PGME) and the training of competent physicians. The purpose of our study was to gain insights from those involved in implementing CBME in two residency programs to inform ongoing implementation practices.

Methods

We conducted a qualitative descriptive study to explore the perspectives of multiple stakeholders involved in the implementation of CBME in two residency programs (the first cohort) to launch the Royal College's Competence by Design model at one Canadian university. Semi-structured interviews were conducted with 17 participants across six stakeholder groups including residents, department chairs, program directors, faculty, medical educators, and program administrators. Data collection and analysis were iterative and reflexive to enhance the authenticity of the results.

Results

The participants' perspectives organized around three key themes including: a) contextualizing curriculum and assessment practices with educational goals of CBME, b) coordinating new administrative requirements to support implementation, and c) adaptability toward a competency-based program structure, each with sub-themes.

Conclusion

By eliciting the perspectives of different stakeholder groups who experienced the implementation processes, we developed a common understanding regarding facilitators and challenges for program directors, program administrators and educational leaders across PGME. Results from our study contribute to the scholarly conversation regarding the key aspects related to CBME implementation and serve to inform its ongoing development and application in various educational contexts.

Operationalizing competency-based assessment: Contextualizing for cultural and gender divides

Following current trends, educational institutions often decide to use a competency framework as an overarching structure in their assessment system. Despite the presence of a common understanding of how different examinations can contribute to the decision on attaining a particular competency, a detailed mapping of the data points appears to be a challenging area that remains to be explored. Faced with the newly emerged task of introducing the assessment of the attainment of UAE medical students against the EmiratesMEDs competency framework, Dubai Medical College for Girls (DMCG) attempted to operationalise the designed concept in the assessment system considering the cultural and gender divide. We believe that health professionals who attempt to implement contextualized competency-based assessment could benefit from being acquainted with our experience. The article offers a step-by-step guide on contextualized competency assessment operationalization, describing building the team, working with consultants and faculty development, estimating institutional assessment capacity, mapping and operationalizing the maps by using both human recourses and the software. We also offer the readers the list of enabling factors and introduce the scope of limitations in the process of developing the competency-based assessment system. We believe that following the present guide can allow educators to operationalize competency-based assessment in any context with respect to local culture and traditions.

A qualitative study of Canadian resident experiences with Competency-Based Medical Education

Background

Competency-based medical education (CBME) is an outcomes-based curricular paradigm focused on ensuring that graduates are competent to meet the needs of patients. Although resident engagement is key to CBME's success, few studies have explored how trainees have experienced CBME implementation. We explored the experiences of residents in Canadian training programs that had implemented CBME.

Methods

We conducted semi-structured interviews with 16 residents in seven Canadian postgraduate training programs, exploring their experiences with CBME. Participants were equally divided between family medicine and specialty programs. Themes were identified using principles of constructivist grounded theory.

Results

Residents were receptive to the goals of CBME, but in practice, described several drawbacks primarily related to assessment and feedback. For many residents, the significant administrative burden and focus on assessment led to performance anxiety. At times, residents felt that assessments lacked meaning as supervisors focused on "checking-boxes" or provided overly broad, non-specific comments. Furthermore, they commonly expressed frustration with the perceived subjectivity and inconsistency of judgments on assessments, especially if assessments were used to delay progression to greater independence, contributing to attempts to "game the system." Faculty engagement and support improved resident experiences with CBME.

Conclusion

Although residents value the potential for CBME to improve the quality of education, assessment and feedback, the current operationalization of CBME may not be consistently achieving these objectives. The authors suggest several initiatives to improve how residents experience assessment and feedback processes in CBME.

Rivalries for attention: insights from a realist evaluation of a postgraduate competency-based medical education implementation in Canada

BACKGROUND

Implementing competency-based medical education (CBME) in post-graduate medical education (PGME) is a complex process that requires multiple systemic changes in a complex system that is simultaneously engaged in multiple initiatives. These initiatives often compete for attention during the implementation of CBME and produce unintended and unanticipated consequences. Understanding the impact of this context is necessary for evaluating the effectiveness of CBME. The purpose of the study was to identify factors, such as contexts and processes, that contribute to the implementation of CBME.

METHODS

We conducted a realist evaluation using data collected from 15 programs through focus groups with residents (2 groups, n = 16) and faculty (one group, n = 8), and semi-structured interviews with program directors (n = 18), and program administrators (n = 12) from 2018 to 2021. Data were analyzed using a template analysis based on a coding framework that was developed from a sample of transcripts, the context-mechanism-outcomes framework for realist evaluations, and the core components of CBME.

RESULTS

The findings demonstrate that simultaneous initiatives in the academic health sciences system creates a key context for CBME implementation - rivalries for attention - and specifically, the introduction of curricular management systems (CMS) concurrent to, but separate from, the implementation of CBME. This context influenced participants' participation, communication, and adaptation during CBME implementation, which led to change fatigue and unmet expectations for the collection and use of assessment data.

CONCLUSIONS

Rival initiatives, such as the concurrent implementation of a new CMS, can have an impact on how programs implement CBME and greatly affect the outcomes of CBME. Mitigating the effects of rivals for attention with flexibility, clear communication, and training can facilitate effective implementation of CBME.