An adaptation-focused evaluation of Canada's first competency-based medical education implementation in radiology

Radiologist preferences for faculty development initiatives to improve resident feedback in the era of competency-based medical education

INTRODUCTION

Since 2022, all Canadian post-graduate medical programs have transitioned to a Competence by Design (CBD) model within a Competency-Based Medical Education (CBME) framework. The CBME model emphasized more frequent, formative assessment of residents to evaluate their progress towards predefined competencies in comparison to traditional medical education models. Faculty members therefore have increased responsibility for providing assessments to residents on a more regular basis, which has associated challenges. Our study explores faculty assessment behaviours within the CBD framework and assesses their openness to opportunities aimed at improving the quality of written feedback. Specifically, we explore faculty's receptiveness to routine metric performance reports that offer comprehensive feedback on their assessment patterns.

METHODS

Online surveys were distributed to all 28 radiology faculty at Queen's University. Data were collected on demographics, feedback practices, motivations for improving the teacher-learner feedback exchange, and openness to metric performance reports and quality improvement measures. Following descriptive statistics, unpaired t-tests and one-way analysis of variance were conducted to compare groups based on experience and subspecialty.

RESULTS

The response rate was 89% (25/28 faculty). 56% of faculty were likely to complete evaluations after working with a resident. Regarding the degree to which faculty felt written feedback is important, 62% found it at least moderately important. A majority (67%) believed that performance reports could influence their evaluation approach, with volume of written feedback being the most likely to change. Faculty expressed interest in feedback-focused development opportunities (67%), favouring Grand Rounds and workshops.

CONCLUSION

Assessment of preceptor perceptions reveals that faculty recognize the importance of offering high-quality written feedback to learners. Faculty openness to quality improvement interventions for curricular reform relies on having sufficient time, knowledge, and skills for effective assessments. This suggests that integrating routine performance metrics into faculty assessments could serve as a catalyst for enhancing future feedback quality.

Competency based medical education implementation at the institutional level: A cross-discipline comparative program evaluation

INTRODUCTION

As an early adopter of competency-based medical education (CBME) our postgraduate institution was uniquely positioned to analyze implementation experience data across programs, while keeping institutional factors constant. We described participants' experiences related to CBME implementation across programs derived from early program evaluation efforts within our setting.

METHODS

This evaluation focused on eight residency programs at a medium-sized academic institution in Canada. Participants (n = 175) included program leaders, faculty, and residents. The study consisted of 3 phases: (1) describing intended implementation; (2) documenting enacted implementation; and (3) comparing intended with enacted implementation to inform adaptations. Each program's findings were summarized in technical reports which were then analyzed thematically. Cross program data were organized by themes.

RESULTS

Six themes were identified. All groups emphasized the need for ongoing refinement of CBME resulting from shared tensions such as increased assessment burden. However, there were some disparate CBME-related experiences between programs such as the experience with entrustable professional activities, the interpretation of retrospective entrustment anchors, and quality of feedback.

CONCLUSION

We detected several cross-program successes and important challenges related to CBME. Our experience can inform other programs engaging in implementation and evaluation of CBME.

Was it all worth it? A graduating resident perspective on CBME

BACKGROUND

Our institution simultaneously transitioned all postgraduate specialty training programs to competency-based medical education (CBME) curricula. We explored experiences of CBME-trained residents graduating from five-year programs to inform the continued evolution of CBME in Canada.

METHODS

We utilized qualitative description to explore residents' experiences and inform continued CBME improvement. Data were collected from fifteen residents from various specialties through focus groups, interviews, and written responses. The data were analyzed inductively, using conventional content analysis.

RESULTS

We identified five overarching themes. Three themes provided insight into residents' experiences with CBME, describing discrepancies between the intentions of CBME and how it was enacted, challenges with implementation, and variation in residents' experiences. Two themes - adaptations and recommendations - could inform meaningful refinements for CBME going forward.

CONCLUSIONS

Residents graduating from CBME training programs offered a balanced perspective, including criticism and recognition of the potential value of CBME when implemented as intended. Their experiences provide a better understanding of residents' needs within CBME curricula, including greater balance and flexibility within programs of assessment and curricula. Many challenges that residents faced with CBME could be alleviated by greater accountability at program, institutional, and national levels. We conclude with actionable recommendations for addressing residents' needs in CBME.

The lived experience of Competence by Design: Canadian resident physicians' perspectives

PURPOSE

Canadian specialist residency programs are in the process of transitioning to a hybrid time and competence model, Competence by Design (CBD), developed by the Royal College of Physicians and Surgeons Canada. Although there is extensive literature around competency-based medical education (CBME), few studies have evaluated the experience of residents after CBME implementation. The purpose of this study was to obtain a rich perspective on the lived experience of residents.

METHODS

We designed a qualitative study with inductive thematic analysis of semistructured interview data. The study population was residents in CBD postgraduate training programs in anesthesiology, internal medicine, or surgery (including all surgical subspecialties) at Dalhousie University (Halifax, NS, Canada).

RESULTS

Residents identified the following benefits of their programs and CBD: supportive peers and clinical supervisors, a roadmap for residency, formalized feedback opportunities, and program evolution. Resident-identified drawbacks of CBD included: a lack of transparency around CBD, CBD not as advertised, a lack of buy-in, increased administrative burden, difficulties obtaining evidence for entrustable professional activities (EPAs); the onus for CBD on residents, inconsistent feedback, cumbersome technology, and significant psychological burden. Resident-suggested improvements were reducing the number of EPAs, streamlining EPA requirements, increasing transparency and communication with competence committees, providing incentives and continuous education for clinical supervisors, improving on existing electronic interfaces, and developing technology better suited to the needs of CBD.

CONCLUSION

This study highlights that the significant administrative and psychological burden of CBD detracts from clinical learning and enthusiasm for residency. Future research could explore whether overcoming the identified challenges will improve residents' experiences.

The Assessment Burden in Competency-Based Medical Education: How Programs Are Adapting

Residents and faculty have described a burden of assessment related to the implementation of competency-based medical education (CBME), which may undermine its benefits. Although this concerning signal has been identified, little has been done to identify adaptations to address this problem. Grounded in an analysis of an early Canadian pan-institutional CBME adopter's experience, this article describes postgraduate programs' adaptations related to the challenges of assessment in CBME. From June 2019-September 2022, 8 residency programs underwent a standardized Rapid Evaluation guided by the Core Components Framework (CCF). Sixty interviews and 18 focus groups were held with invested partners. Transcripts were analyzed abductively using CCF, and ideal implementation was compared with enacted implementation. These findings were then shared back with program leaders, adaptations were subsequently developed, and technical reports were generated for each program. Researchers reviewed the technical reports to identify themes related to the burden of assessment with a subsequent focus on identifying adaptations across programs. Three themes were identified: (1) disparate mental models of assessment processes in CBME, (2) challenges in workplace-based assessment processes, and (3) challenges in performance review and decision making. Theme 1 included entrustment interpretation and lack of shared mindset for performance standards. Adaptations included revising entrustment scales, faculty development, and formalizing resident membership. Theme 2 involved direct observation, timeliness of assessment completion, and feedback quality. Adaptations included alternative assessment strategies beyond entrustable professional activity forms and proactive assessment planning. Theme 3 related to resident data monitoring and competence committee decision making. Adaptations included adding resident representatives to the competence committee and assessment platform enhancements. These adaptations represent responses to the concerning signal of significant burden of assessment within CBME being experienced broadly. The authors hope other programs may learn from their institution's experience and navigate the CBME-related assessment burden their invested partners may be facing.

Exploring the Inner Lens: Examining Oncology Consultants' Use of Competency and Assessment Frameworks in Clinical Teaching Settings

Medical education in oncology has adopted a competency-based medical education (CBME) approach in Canada. Oncologist competencies are arranged within assessment frameworks, but it is unknown whether oncologists routinely use assessment frameworks when determining what to teach oncology trainees. Understanding oncologists' actual approaches to assessment and teaching as a form of enacted CBME could inform and focus faculty development efforts. Using a participatory action research approach, oncology faculty were interviewed regarding their teaching practices and use of assessment frameworks. Faculty suggestions regarding challenges in setting teaching goals and completing assessment within the CBME paradigm were solicited. Furthermore, teaching objectives developed by faculty in-house were assessed regarding the domains of competence targeted. Thirty-one of the 40 faculty members in the Department of Oncology, Western University, agreed to participate. Trainee factors (training level, areas of interest), faculty personal values and experience, the teaching setting, and some components of frameworks (medical expert, communicator, advocate, and professional) were articulated as determinants of teaching goals. Of the 1117 in-house objectives reviewed, 76% focused on the medical role. Faculty held widely disparate views on assessment framework features which would enable teaching and assessment. Oncology faculty members used a multifaceted approach to determine what to teach trainees. This approach contains elements both of the conventional apprenticeship models under which many faculty members themselves were trained and of the CBME paradigm. Future research should focus on exploring ways to resolve the tension between established individual teaching approaches and nationally standardized competency frameworks which are being implemented.