Living in a World of Change: Bridging the Gap From Competency-Based Medical Education Theory to Practice in Canada

Was it all worth it? A graduating resident perspective on CBME

BACKGROUND

Our institution simultaneously transitioned all postgraduate specialty training programs to competency-based medical education (CBME) curricula. We explored experiences of CBME-trained residents graduating from five-year programs to inform the continued evolution of CBME in Canada.

METHODS

We utilized qualitative description to explore residents' experiences and inform continued CBME improvement. Data were collected from fifteen residents from various specialties through focus groups, interviews, and written responses. The data were analyzed inductively, using conventional content analysis.

RESULTS

We identified five overarching themes. Three themes provided insight into residents' experiences with CBME, describing discrepancies between the intentions of CBME and how it was enacted, challenges with implementation, and variation in residents' experiences. Two themes - adaptations and recommendations - could inform meaningful refinements for CBME going forward.

CONCLUSIONS

Residents graduating from CBME training programs offered a balanced perspective, including criticism and recognition of the potential value of CBME when implemented as intended. Their experiences provide a better understanding of residents' needs within CBME curricula, including greater balance and flexibility within programs of assessment and curricula. Many challenges that residents faced with CBME could be alleviated by greater accountability at program, institutional, and national levels. We conclude with actionable recommendations for addressing residents' needs in CBME.

Picking Apart a Program Evaluation Committee: A Multiple Case Study Characterizing Primary Care Residency Program Evaluation Committee Structure, Program Improvement, and Outcomes

BACKGROUND

As of 2014, the Accreditation Council for Graduate Medical Education (ACGME) mandates initiating a Program Evaluation Committee (PEC) to guide ongoing program improvement. However, little guidance nor published reports exist about how individual PECs have undertaken this mandate.

OBJECTIVE

To explore how four primary care residency PECs configure their committees, review program goals and undertake program evaluation and improvement.

METHODS

We conducted a multiple case study between December 2022 and April 2023 of four purposively selected primary care residencies (e.g., family medicine, pediatrics, internal medicine). Data sources included semi-structured interviews with four PEC members per program and diverse program artifacts. Using a constructivist approach, we utilized qualitative coding to analyze participant interviews and content analysis for program artifacts. We then used coded transcripts and artifacts to construct logic models for each program guided by a systems theory lens.  Results: Programs adapt their PEC structure, execution, and outcomes to meet short- and long-term needs based on organizational and program-unique factors such as size and local practices. They relied on multiple data sources and sought diverse stakeholder participation to complete program evaluation and improvement. Identified deficiencies were often categorized as internal versus external to delineate PEC responsibility, boundaries, and feasibility of interventions.

CONCLUSION

The broad guidance provided by the ACGME for PEC configuration allows programs to adapt the committee based on individual needs. However, further instruction on program evaluation and organizational change principles would augment existing PEC efforts.

Effect evaluation of competency-based education (CBE) combined with multi-disciplinary team (MDT) teaching mode in respiratory rehabilitation nursing teaching：A randomized controlled trial

AIM

To explore the application effect of competency-based education (CBE) combined with multi-disciplinary team (MDT) teaching mode in respiratory rehabilitation nursing teaching.

BACKGROUND

Respiratory rehabilitation, as an important non drug treatment for chronic respiratory disease, started late in China, with low implementation rate in hospitals and insufficient knowledge of nursing staff. Therefore, it is urgent to cultivate high professional level nurses with strong professional ability.

DESIGN

A randomized controlled trial.

METHODS

We selected nurses from the respiratory and critical care department at a Grade III, Level A hospital in Beijing, who participated in the respiratory rehabilitation training program between March 2020 and August 2022, as our study participants. Participants were grouped using a random number method. Nurses who participated in the study from March 2020 to May 2021 were set as the control group, and the nurses who participated in the study from June 2021 to August 2022 were set as the test group. The control group used traditional teaching methods, and the test group used the CBE combined MDT teaching model. At the end of the test, the theory, operating skills, satisfaction and core competence were assessed.

RESULTS

After training, the theoretical examination score of the test group (93.71 ± 1.94) was higher than that of the control group (92.37 ± 1.92), the operational examination score of the test group (93.11 ± 2.12) was higher than that of the control group (91.61 ± 1.93), the overall teaching satisfaction of the test group (4.45 ± 0.50) was higher than that of the control group (4.13 ± 0.57), and the total score of the core competence of the test group (148.73 ± 7.08) was higher than that of the control group (141.02 ± 6.41). The difference between the two groups was statistically significant (P<0.05).

CONCLUSION

CBE combined with MDT teaching mode has a good effect in Pulmonary and Critical Care Medicine (PCCM) respiratory rehabilitation nursing teaching.

Lessons learned spanning 17 years of experience with three consecutive nationwide competency based medical education training plans

Introduction

Curricula for postgraduate medical education have transformed since the introduction of competency based medical education (CBME). Postgraduate training plans offer broader training with different competencies and an outcome-based approach, in addition to the medical technical aspects of training. However, CBME also has its challenges. Over the past years, critical views have been shared on the potential drawbacks of CBME, such as assessment burden and conflicts with practicality in the workplace. Recent studies identified a need for a better understanding of how the evolving concept of CBME has been translated to curriculum design and implemented in the practice of postgraduate training. The aim of this study was to describe the development of CBME translations to curriculum design, based on three consecutive postgraduate training programs spanning 17 years.

Method

We performed a document analysis of three consecutive Dutch gynecology and obstetrics training plans that were implemented in 2005, 2013, and 2021. We used template analysis to identify changes over time.

Results

Over time, CBME-based curriculum design changed in several domains. Assessment changed from a model with a focus on summative decision to one with an emphasis on formative, low-stakes assessments aimed at supporting learning. The training plans evolved in parallel to evolving educational insights, e.g., by placing increasing emphasis on personal development. The curricula focused on a competency-based concept by introducing training modules and personalized authorization based on feedback rather than on a set duration of internships. There was increasing freedom in personalized training trajectories in the training plans, together with increasing trust towards the resident.

Conclusion

The way CBME was translated into training plans has evolved in the course of 17 years of experience with CMBE-based education. The main areas of change were the structure of the training plans, which became increasingly open, the degree to which learning outcomes were mandatory or not, and the way these outcomes were assessed.

Education modalities for serious illness communication training: A scoping review on the impact on clinician behavior and patient outcomes

BACKGROUND

Several clinician training interventions have been developed in the past decade to address serious illness communication. While numerous studies report on clinician attitudes and confidence, little is reported on individual education modalities and their impact on actual behavior change and patient outcomes.

AIM

To examine what is known about the education modalities used in serious illness communication training and their impact on clinician behaviors and patient outcomes.

DESIGN

A scoping review using the Joanna Briggs Methods Manual for Scoping Reviews was conducted to examine studies measuring clinician behaviors or patient outcomes.

DATA SOURCES

Ovid MEDLINE and EMBASE databases were searched for English-language studies published between January 2011 and March 2023.

RESULTS

The search identified 1317 articles: 76 met inclusion criteria describing 64 unique interventions. Common education modalities used were: single workshop (n = 29), multiple workshops (n = 11), single workshop with coaching (n = 7), and multiple workshops with coaching (n = 5); though they were inconsistently structured. Studies reporting improved clinician skills tended to be in simulation settings with neither clinical practice nor patient outcomes explored. While some studies reported behavior changes or improved patient outcomes, they did not necessarily confirm improvements in clinician skills. As multiple modalities were commonly used and often embedded within quality improvement initiatives, the impact of individual modalities could not be determined.

CONCLUSION

This scoping review of serious illness communication interventions found heterogeneity among education modalities used and limited evidence supporting their effectiveness in impacting patient-centered outcomes and long-term clinician skill acquisition. Well-defined educational modalities and consistent measures of behavior change and standard patient-centered outcomes are needed.

Using a rapid-cycle approach to evaluate implementation of competency-based medical education in ophthalmology

OBJECTIVE

As competency-based medical education is being implemented across Canada, there is an increasing need to evaluate the progress to date, including identification of strengths and weaknesses, to inform program development. Ophthalmology is preparing for a national launch in coming years. The purpose of this study was to describe key stakeholders' lived experiences in the competency-based medical education foundation-of-discipline stage in one ophthalmology department.

DESIGN

Using a case-study approach, a qualitative rapid-cycle evaluation was conducted during the 2018-2019 academic year.

PARTICIPANTS

Residents, faculty, academic advisors, competence committee members, the program director, the program administrator, and the educational consultant were invited to participate in the program evaluation.

METHODS

The rapid-cycle evaluation consisted of 2 evaluation cycles, with the first round of interviews and focus groups occurring in October 2018 and the second round in March 2019. Recommendations were implemented in November 2019 and June 2019. All data were analyzed thematically using NVivo.

RESULTS

Three main themes emerged across all data sets: developing a shared understanding (e.g., role expectations and changes to assessment), refining assessment processes and tools (e.g., the need for streamlining and clarification), and feedback (e.g., perceived benefits and value of narrative comments).

CONCLUSIONS

Exploring lived experiences in this study resulted in positive and immediate improvements to the residency program. The recommendations and approach will be useful to other Canadian departments and institutions as they prepare for Competence by Design.

Exploring stakeholder perspectives regarding the implementation of competency-based medical education: a qualitative descriptive study

Introduction

Competency-based medical education (CBME) offers perceived advantages and benefits for postgraduate medical education (PGME) and the training of competent physicians. The purpose of our study was to gain insights from those involved in implementing CBME in two residency programs to inform ongoing implementation practices.

Methods

We conducted a qualitative descriptive study to explore the perspectives of multiple stakeholders involved in the implementation of CBME in two residency programs (the first cohort) to launch the Royal College's Competence by Design model at one Canadian university. Semi-structured interviews were conducted with 17 participants across six stakeholder groups including residents, department chairs, program directors, faculty, medical educators, and program administrators. Data collection and analysis were iterative and reflexive to enhance the authenticity of the results.

Results

The participants' perspectives organized around three key themes including: a) contextualizing curriculum and assessment practices with educational goals of CBME, b) coordinating new administrative requirements to support implementation, and c) adaptability toward a competency-based program structure, each with sub-themes.

Conclusion

By eliciting the perspectives of different stakeholder groups who experienced the implementation processes, we developed a common understanding regarding facilitators and challenges for program directors, program administrators and educational leaders across PGME. Results from our study contribute to the scholarly conversation regarding the key aspects related to CBME implementation and serve to inform its ongoing development and application in various educational contexts.

Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program

BACKGROUND

Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design.

METHODS

We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program's use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs.

RESULTS

Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a "numbers game" residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning.

CONCLUSION

Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training.

Overcoming the barriers to implementation of competence-based medical education in post-graduate medical education: a narrative literature review

To ensure that residents are equipped with the necessary skills for practice, competence-based medical education (CBME) represents a transformative change in postgraduate medical education, which is being progressively introduced across Canadian specialty residency programs. Successful implementation will require adjustments to curriculum, assessment, and evaluation, with careful attention to the unique needs in the local context, including resident and faculty development. This narrative review of the literature aimed to determine the potential barriers to the successful implementation of CBME and the strategies by which they can be addressed, with a specific consideration of the author's program in pediatrics in Manitoba. Eleven articles were identified with a specific focus on the implementation of CBME in the post-graduate setting, and 10 were included in the review after critical appraisal. Three key themes emerged from the articles: the value of broad stakeholder engagement and leadership, the importance of faculty and resident development, and the development of specific support systems for the educational curriculum. Different strategies were considered and contrasted for addressing these important themes. This review provides important insights and practical approaches to the barriers that should be useful as programs prepare for the implementation of CBME.

Learning from the implementation of clinical empathy training: an explorative qualitative study in search of the barriers and facilitators

BACKGROUND

Amid concerns about the decline of empathy during the clinical training of medical clerks, evidence that empathy improves patient outcomes suggests some potential for teaching empathy in ways that will affect the knowledge, attitude and behaviour of medical clerks. This potential alone cannot, however, guarantee the success of educational innovations to introduce empathy to the medical curriculum. This research aims to identify the barriers and facilitators of the implementation of a specific clinical initiative to enhance the empathy skills of clerks, namely the training of clerks to act as a 'MedGezel' or 'medical coach'.

METHOD

We conducted an explorative qualitative study based on interview data collected and analyzed using reflexive thematic analysis and the readiness for change theory. We conducted semi-structured interviews with relevant stakeholders in this particular qualitative study. Thematic analysis was based on open and axial coding using ATLAS.ti 9, which facilitated the emergence of common themes of interest and meaning for the study.

RESULTS

A total of 13 relevant stakeholders participated as interviewees in our study. The data was collected from April to June 2021. Our analysis generated 6 main themes which can provide insights into why the implementation of the MedGezel educational innovation failed so far. The following themes emerged: the case for change: why change?; practical necessity; leadership; management and resources; staff culture; and alignment with the corporate strategy.

DISCUSSION

The implementation failure can be partially explained as resulting from the personal attitudes and choices of participants, who struggled to reconcile a vision that they liked with side effects that they feared. While participants repeatedly mentioned management and leadership issues, these organizational issues seemed less important as they could be easily resolved in practice. What was more important and fatal for the initiative was its lack of alignment with staff culture, despite its alignment with corporate strategy.

CONCLUSION

This investigation into the barriers and facilitators influencing the implementation of the MedGezel program identified 6 explanatory themes, the most impactful one being staff culture.

Exploring the relationship between social accountability and competency-based medical education: A narrative review

BACKGROUND

Social accountability (SA), a quintessential goal of medical education, has been discussed as a precipitant for the transition toward competency-based medical education (CBME). However, the relationship between SA and CBME remains unclear. A narrative review was conducted to systematically explore the relationship between SA and CBME as described in the literature.

METHODS

Electronic databases, select journals, and medical education organizations were systematically searched. 363 titles and abstracts were screened and 147 full texts were reviewed. The salient text was extracted from 36 records, which were then inductively coded before narrative synthesis and interpretation.

RESULTS

The relationship between SA and CBME was described in three manners: (1) CBME as a natural driver of SA where CBME was perceived to be inherently socially accountable, (2) CBME as an opportunistic mechanism for actively changing medical training to better meet standards of SA, and (3) CBME as a tool to measure SA relating to measurable outcomes data provided by CBME.

CONCLUSION

CBME has theoretical potential to assist programs in becoming more socially accountable if the communities they serve are considered key stakeholders in the design, implementation, and evaluation. A paucity of evidence remains which provides empirical evidence of SA within programs that have implemented CBME.

Developing a Competency Framework of Interprofessional Occupational Health Team: A First Step to Interprofessional Education in Occupational Health Field

OBJECTIVE

The present study aimed to develop a competency framework for interprofessional collaboration in an occupational health team.

METHODS

This study was carried out in three stages. In the first stage, the viewpoints of different stakeholders on the essential competencies of interprofessional team members in occupational health services (OHS) were assessed. In the second stage, extracted codes were converted into competencies in expert panels. After that, the extracted items were categorized by inductive content analysis approach into competency framework. In the third stage, the content validity of interprofessional competencies in OHS was evaluated.

RESULTS

The competency framework was finalized in six competency domains including prevention, occupational disease management, occupational health assessments, inter-disciplinary and interprofessional collaboration, education, and evidence-based practice and professionalism.

CONCLUSION

The framework could be developed in interprofessional education in OHS based on competency-based education approach.

Can case-based discussions in a group setting be used to assess residents' clinical skills?

OBJECTIVES

The purpose of this study was to explore residents' and assessors' perception of a new group assessment concept.

METHODS

This qualitative study consists of observations of four group assessment sessions, followed by semi-structured interviews with six residents and four assessors (specialists in internal medicine), who all volunteered to be interviewed. All residents at a medical department (eleven to fifteen each time) and four assessors participated in four group assessments, where the residents' clinical skills were assessed through case-based discussions. An external consultant (an anthropologist) performed the observations and the interviews. Notes from the observations and the interviews were analyzed using an inductive approach.

RESULTS

Eight of the ten interviewed participants preferred group assessment to individual assessment. Results from the interviews suggested that the group assessments were more consistent and that the level of discussion was perceived to be higher in the group discussions compared to the one-to-one discussions. All residents indicated that they had acquired new knowledge during their assessment and reported having learned from listening to the assessment of their peers. Assessors similarly reported gaining new knowledge.

CONCLUSIONS

The residents and assessors expressed very favourable attitudes toward the new group assessment concept. The assessment process was perceived to be higher in quality and more consistent, contributing to learning for all participating doctors in the department. Group assessment is feasible and acceptable, and provides a promising tool for assessment of clinical skills in the future.

Creating a Competency-Based Medical Education Curriculum for Canadian Diagnostic Radiology Residency (Queen's Fundamental Innovations in Residency Education)-Part 2: Core of Discipline Stage

PURPOSE

All postgraduate residency programs in Canada are transitioning to a competency-based medical education (CBME) model divided into 4 stages of training. Queen's University has been the first Canadian institution to mandate transitioning to CBME across all residency programs, including Diagnostic Radiology. This study describes the implementation of CBME with a focus on the third developmental stage, Core of Discipline, in the Diagnostic Radiology residency program at Queen's University. We describe strategies applied and challenges encountered during the adoption and implementation process in order to inform the development of other CBME residency programs in Diagnostic Radiology.

METHODS

At Queen's University, the Core of Discipline stage was developed using the Royal College of Physicians and Surgeons of Canada's (RCPSC) competence continuum guidelines and the CanMEDS framework to create radiology-specific entrustable professional activities (EPAs) and milestones for assessment. New committees, administrative positions, and assessment strategies were created to develop these assessment guidelines. Currently, 2 cohorts of residents (n = 6) are enrolled in the Core of Discipline stage.

RESULTS

EPAs, milestones, and methods of evaluation for the Core of Discipline stage are described. Opportunities during implementation included tracking progress toward educational objectives and increased mentorship. Challenges included difficulty meeting procedural volume requirements, inconsistent procedural tracking, improving feedback mechanisms, and administrative burden.

CONCLUSION

The transition to a competency-based curriculum in an academic Diagnostic Radiology residency program is significantly resource and time intensive. This report describes challenges faced in developing the Core of Discipline stage and potential solutions to facilitate this process.

Translating outcome frameworks to assessment programmes: Implications for validity

OBJECTIVES

Competency-based medical education (CBME) requires that educators structure assessment of clinical competence using outcome frameworks. Although these frameworks may serve some outcomes well (e.g. represent eventual practice), translating these into workplace-based assessment plans may undermine validity and, therefore, trustworthiness of assessment decisions due to a number of competing factors that may not always be visible or their impact knowable. Explored here is the translation process from outcome framework to formative and summative assessment plans in postgraduate medical education (PGME) in three Canadian universities.

METHODS

We conducted a qualitative study involving in-depth semi-structured interviews with leaders of PGME programmes involved in assessment and/or CBME implementation, with a focus on their assessment-based translational activities and evaluation strategies. Interviews were informed by Callon's theory of translation. Our analytical strategy involved directed content analysis, allowing us to be guided by Kane's validity framework, whilst still participating in open coding and analytical memo taking. We then engaged in axial coding to systematically explore themes across the dataset, various situations and our conceptual framework.

RESULTS

Twenty-four interviews were conducted involving 15 specialties across three universities. Our results suggest: (i) using outcomes frameworks for assessment is necessary for good assessment but are also viewed as incomplete constructs; (ii) there are a number of social and practical negotiations with competing factors that displace validity as a core influencer in assessment planning, including implementation, accreditation and technology; and (iii) validity exists as threatened, uncertain and assumed due to a number of unchecked assumptions and reliance on surrogates.

CONCLUSIONS

Translational processes in CBME involve negotiating with numerous influencing actors and institutions that, from an assessment perspective, provide challenges for assessment scientists, institutions and educators to contend with. These processes are challenging validity as a core element of assessment designs. Educators must reconcile these influences when preparing for or structuring validity arguments.