The Impact of a National Competency-Based Medical Education Initiative in Family Medicine

Evaluating competency-based medical education: a systematized review of current practices

BACKGROUND

Few published articles provide a comprehensive overview of the available evidence on the topic of evaluating competency-based medical education (CBME) curricula. The purpose of this review is therefore to synthesize the available evidence on the evaluation practices for competency-based curricula employed in schools and programs for undergraduate and postgraduate health professionals.

METHOD

This systematized review was conducted following the systematic reviews approach with minor modifications to synthesize the findings of published studies that examined the evaluation of CBME undergraduate and postgraduate programs for health professionals.

RESULTS

Thirty-eight articles met the inclusion criteria and reported evaluation practices in CBME curricula from various countries and regions worldwide, such as Canada, China, Turkey, and West Africa. 57% of the evaluated programs were at the postgraduate level, and 71% were in the field of medicine. The results revealed variation in reporting evaluation practices, with numerous studies failing to clarify evaluations' objectives, approaches, tools, and standards as well as how evaluations were reported and communicated. It was noted that questionnaires were the primary tool employed for evaluating programs, often combined with interviews or focus groups. Furthermore, the utilized evaluation standards considered the well-known competencies framework, specialized association guidelines, and accreditation criteria.

CONCLUSION

This review calls attention to the importance of ensuring that reports of evaluation experiences include certain essential elements of evaluation to better inform theory and practice.

A roadmap to realist interviews in health professions education research: Recommendations based on a critical analysis

CONTEXT

Realist evaluation is increasingly employed in health professions education research (HPER) because it can unpack the extent to which complex educational interventions work (or not), for whom under what circumstances and how. While realist evaluation is not wedded to particular methods, realist interviews are commonly the primary, if not only, data collection method in realist evaluations. While qualitative interviewing from an interpretivist standpoint has been well-articulated in the HPER literature, realist interviewing differs substantially. The former elicits participants' views and experiences of a topic of inquiry, whereas realist interviewing focuses on building, testing and/or refining programme theory. Therefore, this article aims to help readers better understand, conduct, report and critique realist interviews as part of realist evaluations.

METHODS

In this paper, we describe what realist approaches are, what realist interviewing is and why realist interviewing matters. We outline five stages to realist interviewing (developing initial programme theory, realist sampling/samples, the interview itself, realist analysis and reporting realist interviews), drawing on two illustrative cases from our own realist evaluations employing interviewing to bring theory to life. We provide a critical analysis of 12 realist evaluations employing interviewing in the HPER literature. Alongside reporting standards, and our own realist interviewing experiences, this critical analysis of published articles serves to foreground our recommendations for realist interviewing.

CONCLUSIONS

We encourage HPE researchers to consider realist interviews as part of realist evaluations of complex interventions. Our critical analysis reveals that realist interviews can provide unique insights into HPE, but authors now need to report their sampling approach, type of interviewing and interview questions more explicitly. Studies should also more explicitly draw on existing realist interviewing literature and follow reporting guidelines for realist evaluations. We hope this paper provides a useful roadmap to conducting, reporting and critically appraising realist interviews in HPER.

Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Researching models of innovation and adoption in health professions education

BACKGROUND

Despite the constant presence of change and innovation in health professions education (HPE), there has been relatively little theoretical modelling of such change, the experiences of change, the ideology associated with change or the unexpected consequences of change. In this paper, the authors explore theoretical approaches to the adoption of innovations in HPE as a way of mapping a broader theoretical landscape of change.

METHOD

The authors, HPE researchers with an interest in technology adoption and systemic change, present a narrative review of the literature based on a series of thought experiments regarding how communities and individuals respond to the introduction of new ideas or methods. This research investigates the stages of innovation adoption, from the emergence and hype around new ideas to the concrete experiences of early adopters.

RESULTS

When an innovation first emerges, there is often little concrete information available to inform potential adopters, leaving it susceptible to hype, both positive and negative. This can be described using the Gartner Hype Cycle model, albeit with important caveats. Once the adoption of an innovation gets underway, early adopter user experiences can inform those that follow. This can be described using Rogers' diffusion of innovation model, again with caveats. Notably, neither model goes beyond the point of single point-in-time, yes/no, individual adoption. Other approaches, such as learning curve theory, are needed to track uptake and maintenance by individuals over time.

SIGNIFICANCE

This expanded theoretical base, while still somewhat instrumentalist, combined with complementary theoretical perspectives can afford opportunities to better explore reasons for variance, volunteerism and resistance to change. In summary, change is complicated and nuanced, and better models and theories are needed to understand and work meaningfully with change in HPE. To that end, the authors seek to encourage richer and more thoughtful research and scholarly thinking about change and a more nuanced approach to the pursuit of change in HPE as a whole.

Inter-institutional data-driven education research: consensus values, principles, and recommendations to guide the ethical sharing of administrative education data in the Canadian medical education research context

Background

Administrative data are generated when educating, licensing, and regulating future physicians but these data are rarely used beyond their pre-specified purposes. The capacity necessary for sensitive and responsive oversight that supports the sharing of administrative medical education data across institutions for research purposes needs to be developed.

Method

A pan-Canadian consensus-building project was undertaken to develop agreement on the goals, benefits, risks, values, and principles that should underpin inter-institutional data-driven medical education research in Canada. A survey of key literature, consultations with various stakeholders and five successive knowledge synthesis workshops informed this project. Propositions were developed, driving subsequent discussions until collective agreement was distilled.

Results

Consensus coalesced around six key principles: establishing clear purposes, rationale, and methodology for inter-institutional data-driven research a priori; informed consent from data generators in education systems is non-negotiable; multi-institutional data sharing requires special governance; data governance should be guided by data sovereignty; data use should be guided by an identified set of shared values; and best practices in research data-management should be applied.

Conclusion

We recommend establishing a representative governance body, engaging trusted data facility, and adherence to extant data management policies when sharing administrative medical education data for research purposes in Canada.

Context matters in curriculum reform: An analysis of change in surgical training

INTRODUCTION

Education and training reforms are typically devised by accreditation bodies and rolled out nationally. This top-down approach is positioned as contextually independent, yet context is highly influential in shaping the impact of change. Given this, it is critical to consider how curriculum reform plays out as it meets local settings. We have therefore used a national-level curriculum reform process of surgical training, Improving Surgical Training (IST), to examine the influence of context in IST implementation across two UK countries.

METHODS

Adopting a case study approach, we used document data for contextualisation purposes and semi-structured interviews with key stakeholders across multiple organisations (n = 17, plus four follow-up interviews) as our main source of data. Initial data coding and analysis were inductive. We followed this with a secondary analysis using Engeström's second-generation activity theory nested within an overarching framework of complexity theory to help tease out some key elements of IST development and implementation.

RESULTS

The introduction of IST into the surgical training system was historically situated within a landscape of previous reforms. IST's aims collided with existing practices and rules, thus creating tensions. In one country, the systems of IST and surgical training came together to some extent, mostly due to processes of social networks, negotiation and leverage nested in a relatively cohesive setting. These processes were not apparent in the other country, and instead of transformative change, the system contracted. Change was not integrated, and the reform was halted.

CONCLUSIONS

Our use of a case study approach and complexity theory deepens understanding of how history, systems and contexts interact to facilitate or inhibit change within one area of medical education. Our study paves the way for further empirical work examining the influence of context in curriculum reform, and thus determining how best to bring about change in practice.

Are early career family physicians prepared for practice in Canada? A qualitative study

BACKGROUND

In Canada, the College of Family Physicians of Canada (CFPC) introduced Competency Based Medical Education to prepare and train family medicine residents to be competent to enter and adapt to the independent practice of comprehensive family medicine. Despite its implementation, the scope of practice is narrowing. This study aims to understand the degree to which early career Family Physicians (FPs) are prepared for independent practice.

METHOD

A qualitative design was used for this study. A survey and focus groups were conducted with early-career FPs who completed residency training in Canada. The survey and focus groups examined the degree of preparedness of early career FPs in relation to 37 core professional activities identified by the CFPC's Residency Training Profile. Descriptive statistics and qualitative content analysis were conducted.

RESULTS

Seventy-five participants from across Canada participated in the survey, and 59 participated in the focus groups. Early career FPs reported being well prepared to provide continuous and coordinated care for patients with common presentations and deliver various services to different populations. FPs were also well prepared to manage the electronic medical record, participate in team-based care, provide regular and after-hours coverage, and assume leadership and teaching roles. However, FPs reported being less prepared for virtual care, business management, providing culturally safe care, delivering specific services in emergency care hospitals, obstetrics, self-care, engaging with the local communities, and conducting research activities.

CONCLUSIONS

Early career FPs do not feel fully prepared for practice in all 37 core activities in the Residency Training Profile. As part of the introduction of the three-year program by the CFPC, the postgraduate family medicine training should consider providing more exposure to learning opportunities and developing curricula in the areas where FPs are unprepared for practice. These changes could facilitate the production of a FP workforce better prepared to manage the dynamic and complex challenges and dilemmas faced in independent practice.

Development, Implementation, and Meta-Evaluation of a National Approach to Programmatic Assessment in Canadian Family Medicine Residency Training

The growing international adoption of competency-based medical education has created a desire for descriptions of innovative assessment approaches that generate appropriate and sufficient information to allow for informed, defensible decisions about learner progress. In this article, the authors provide an overview of the development and implementation of the approach to programmatic assessment in postgraduate family medicine training programs in Canada, called Continuous Reflective Assessment for Training (CRAFT). CRAFT is a principles-guided, high-level approach to workplace-based assessment that was intentionally designed to be adaptable to local contexts, including size of program, resources available, and structural enablers and barriers. CRAFT has been implemented in all 17 Canadian family medicine residency programs, with each program taking advantage of the high-level nature of the CRAFT guidelines to create bespoke assessment processes and tools appropriate for their local contexts. Similarities and differences in CRAFT implementation between 5 different family medicine residency training programs, representing both English- and French-language programs from both Western and Eastern Canada, are described. Despite the intentional flexibility of the CRAFT guidelines, notable similarities in assessment processes and procedures across the 5 programs were seen. A meta-evaluation of findings from programs that have published evaluation information supports the value of CRAFT as an effective approach to programmatic assessment. While CRAFT is currently in place in family medicine residency programs in Canada, given its adaptability to different contexts as well as promising evaluation data, the CRAFT approach shows promise for application in other training environments.

Fundamental Teaching Activities in Family Medicine Framework: Analysis of Awareness and Utilization

INTRODUCTION

In 2015, the College of Family Physicians of Canada, in performing their commitment to supporting its members in their educational roles, created the Family Medicine Framework (FTA). It was designed to assist family medicine educators with an understanding of the core activities of educators: precepting, coaching, and teaching within or beyond clinical settings. Given that an examination of member awareness of FTA has not been previously undertaken, our primary objective was to conduct an evaluation on its utility and application.

METHODS

In partnership with College of Family Physicians of Canada Faculty Development Education Committee members, we used a practical participatory evaluation approach to conduct a two-phase mixed-methods evaluation of the FTA. We distributed an electronic survey in French and English languages to Canadian faculty development, program, and site directors in family medicine. We then conducted follow-up interviews with self-selected participants.

RESULTS

Of the target populations, 12/15 (80%) faculty development directors (FDDs), 12/18 (66.7%) program directors, and 34/174 (19.5%) site directors completed the electronic survey. Subsequently, 6 FDDs, 3 program directors, and 3 site directors completed an interview (n = 12). Findings indicate that awareness of the FTA was highest among FDDs. Facilitators who encourage teachers to use the FTA and barriers for low uptake were also identified.

DISCUSSION

This evaluation illuminated that varied levels of awareness of the FTA may contribute to the low uptake among education leaders. We also suggest future research to address possible barriers that hinder effective applications of the FTA in faculty development initiatives.

Program evaluation: An educator's portal into academic scholarship

Program evaluation is an "essential responsibility" but is often not seen as a scholarly pursuit. While Boyer expanded what qualifies as educational scholarship, many still need to engage in processes that are rigorous and of a requisite academic standard to be labelled as scholarly. Many medical educators may feel that scholarly program evaluation is a daunting task due to the competing interests of curricular change, remediation, and clinical care. This paper explores how educators can take their questions around outcomes and efficacy of our programs and efficiently engage in education scholarship. The authors outline how educators can examine whether training programs have a desired impact and outcomes, and then how they might leverage this process into education scholarship.

A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education

The coronavirus disease 2019 pandemic resulted in a dramatic change in the Royal College of Physicians and Surgeons of Canada assessment process through elimination of the oral and practical components of the 2020 Anatomical Pathology examination. Our study sought to determine stakeholder opinions and experiences on these changes in the context of the 2019 implementation of competency-based medical education. Surveys were designed for residents and practicing pathologists. In total, 57 residents (estimated response rate 29%) and 185 pathologists (estimated response rate 19%) participated across Canada; 67% of pathologists disagreed with the 2020 Royal College examination changes, compared with 30% for residents (P = <.00001). When asked whether the Royal College examination should be eliminated, 95% of pathologists indicated they would be against this, compared to only 34% of residents (P = <.00001). Perceptions on changes to and importance of different components of assessment in competency-based medical education were similar between pathologists and residents, with participants perceiving assessment practices to have changed fairly little since its implementation, with the exception of more frequent feedback. Analysis of narrative comments identified several common themes around assessment, including the need for objectivity and standardization and the problem of failure-to-fail. However, residents identified numerous elements of their performance that can be assessed only through longitudinal evaluation. Pathologists, on the other hand, tended to view these aspects of performance as laden with bias. Our results will hopefully help guide future innovation in assessment by characterizing different stakeholder perspectives on key issues in medical education.

Tensions in describing competency-based medical education: a study of Canadian key opinion leaders

The current discourse on competency-based medical education (CBME) is confounded by a lack of agreement on definitions and philosophical assumptions. This phenomenon impacts curriculum implementation, program evaluation and disrupts dialogue with the education community. The purpose of this study is to explore how Canadian key opinion leaders describe the philosophy and practice of CBME. A purposeful and snowball sample of Canadian key opinion leaders, reflecting diversity of institutions and academic roles, was recruited. A qualitative thematic analysis of semi-structured interviews was conducted using the principles of constructivist grounded theory. A modified integrated knowledge user checking process was accomplished via a national open meeting of educators, researchers, and leaders in postgraduate medical education. Research ethics board approval was received. 17 interviews were completed between September and November 2018. 43 participants attended the open meeting. There was no unified framing or definition of CBME; perspectives were heterogenous. Most participants struggled to identify a philosophy or theory that underpinned CBME. CBME was often defined by key operational practices, including an emphasis on work-based assessments and coaching relationships between learners and supervisors. CBME was articulated as addressing problems with current training models, including failure to fail, rigor in the structure of training and maintaining the social contract with the public. The unintended consequences of CBME included a reductionist framing of competence and concern for resident wellness with changes to the learning environment. This study demonstrates a heterogeneity in defining CMBE among Canadian key opinion leaders. Future work should explore the fidelity of implementation of CBME.

Trends and challenges of medical education

The aim of this article is to reflect on the contextual factors that influence medical education, in order to provide some recommendations about the challenges that physicians face. A literature review was conducted to write this reflection paper, and it was found that the education of health professionals has undergone significant changes that are based on the development of biomedical sciences, technology, and pedagogy, and are influenced by the needs of the general population, the requirements of the health systems of each country, and scientific progress. These changes have given rise to challenges inherent to the special characteristics of each country and related to the quantity, quality, and relevance of the training of physicians. In this regard, it has been suggested that, in the face of such changes, it is necessary to design curricula that integrate all aspects of health care, consider the supply and demand of medical services, emphasize professionalism, take into account the pedagogical training of health sciences teachers, comply with quality standards, assist students in selecting undergraduate programs adequately (in this case Medicine), and ensure relevant clinical practice scenarios. It should also be noted that medical education has not adapted adequately to the aforementioned changes in many countries and, therefore, the development of teaching and learning strategies has lagged behind and physicians in such countries do not have the capacity to provide the health care required by the general population in a proper manner.

Association Between Family Medicine Residents' Mindsets and In-Training Exam Scores

Introduction

In medical practice, a mastery mindset is important for engaging in lifelong learning. The objective of this study was to examine the association between family medicine residents' scores on mindset measures and their performance on in-training examinations (ITE).

Methods

This was a secondary data analysis of a cohort of family medicine residents. Following ethics approval, residents' ITE scores from each of the 2 years of residency were linked with residents' responses to a mindsets survey that they had taken at the midpoint of residency training. Multiple regression analysis was used to investigate the relationship between residents' mindset scores and their ITE scores. Of 85 residents, 46 (54%) had complete data for the three data collection points.

Results

Residents' ITE scores in year 1 were most predictive of their ITE scores in year 2 (β=0.72; P<.001). Mastery mindset scores were negatively associated with residents' performance on the ITE in year 2 (β=-0.29; P=.004).

Conclusion

While the observed negative relationship between residents' mastery mindset scores and their ITE performance may be disconcerting, it is not surprising. In clinical settings, residents are individually coached by preceptors and provided with specific, actionable feedback to support their learning. With respect to formative assessments, residents likely require explicit training on how to use their assessment results (ITE scores) to support their self-directed learning. This finding has practical implications for residency programs in using ITEs as formative assessments.

Living in a World of Change: Bridging the Gap From Competency-Based Medical Education Theory to Practice in Canada

Within graduate medical education, many educators are experiencing a climate of significant change. One transformation, competency-based medical education (CBME), is occurring simultaneously across much of the world, and implementation will require navigating numerous tensions and paradoxes. Successful transformation requires many types of power and is most likely to happen when the medical education community of professionals is engaged in designing, experimenting, acting, and sensemaking together.In this complex climate, the craft of change facilitators and community leaders is needed more than ever. National top-down policies and structures, while important, are not sufficient. The operationalization of new advances is best done when local leaders are afforded room to shape their local context. An evidence-based approach to thinking about the transformative change associated with CBME needs to be adopted. In this age of entrustment, 3 priorities are paramount: (1) engage, entrust, and empower professionals with increasing shared ownership of the innovation; (2) better prepare education professionals in leadership and transformational change techniques in the complex system of medical education; and (3) leverage the wider community of practice to maximize local CBME customization. These recommendations, although based largely on the Canadian experience, are intended to inform CBME transformation in any context.

Process and outcome evaluation of a CBME intervention guided by program theory

RATIONALE

Competency-based medical education (CBME) has gained momentum as an improved training model, but literature on outcomes of CBME, including evaluation of implementation processes, is minimal. We present a case for the following: (a) the development of a program theory is essential prior to or in the initial stages of implementation of CBME; (b) the program theory should guide the strategies and methods for evaluation that will answer questions about anticipated and unintended outcomes; and (c) the iterative process of testing assumptions and hypotheses will lead to modifications to the program theory to inform best practices of implementing CBME.

METHODS

We use the Triple C Competency-based Curriculum as a worked example to illustrate how process and outcome evaluation, guided by a program theory, can lead to meaningful enhancement of CBME curriculum, assessment, and implementation strategies. Using a mixed methods design, the processes and outcomes of Triple C were explored through surveys, interviews, and historical document review, which captured the experiences of various stakeholders.

FINDINGS

The theory-led program evaluation process was able to identify areas that supported CBME implementation: the value of a strong nondirective national vertical core supporting the transformation in education, program autonomy, and adaptability to pre-existing local context. Areas in need of improvement included the need for ongoing support from College of Family Physicians of Canada (CFPC) and better planning for shifts in program leadership over time.

CONCLUSIONS

Deliberately pairing evaluation alongside change is an important activity and, when accomplished, yields valuable information from the experiences of those implementing and experiencing a program. Evaluation and the development of an updated program theory facilitate the introduction of new changes and theories that build on these findings, which also supports the desired goal of contributing toward cumulative science rather than "reinventing the wheel."

Conceptualization of Competency-Based Medical Education Terminology in Family Medicine Postgraduate Medical Education and Continuing Professional Development: A Scoping Review

PURPOSE

To examine the extent, range, and nature of how competency-based medical education (CBME) implementation terminology is used (i.e., the conceptualization of CBME-related terms) within the family medicine postgraduate medical education (PGME) and continuing professional development (CPD) literature.

METHOD

This scoping review's methodology was based on Arksey and O'Malley's framework and subsequent recommendations by Tricco and colleagues. The authors searched 5 databases and the gray literature for U.S. and Canadian publications between January 2000 and April 2017. Full-text English-language articles on CBME implementation that focused exclusively on family medicine PGME and/or CPD programs were eligible for inclusion. A standardized data extraction form was used to collect article demographic data and coding concepts data. Data analysis used mixed methods, including quantitative frequency analysis and qualitative thematic analysis.

RESULTS

Of 470 unique articles identified, 80 (17%) met the inclusion criteria and were selected for inclusion in the review. Only 12 (15%) of the 80 articles provided a referenced definition of the coding concepts (i.e., referred to an article/organization as the definition's source), resulting in 19 highly variable-and 12 unique- referenced definitions of key terms used in CBME implementation (competence, competency, competency-based medical education). Thematic analysis of the referenced definitions identified 15 dominant themes, among which the most common were (1) a multidimensional and dynamic concept that encompasses a variety of skill components and (2) being able to use communication, knowledge, technical skills, clinical reasoning, judgment, emotions, attitudes, personal values, and reflection in practice.

CONCLUSIONS

The construction and dissemination of shared definitions is essential to CBME's successful implementation. The low number of referenced definitions and lack of consensus on such definitions suggest more attention needs to be paid to conceptual rigor. The authors recommend those involved in family medicine education work with colleagues across medical specialties to develop a common taxonomy.