Changing the culture of medical training: An important step toward the implementation of competency-based medical education

Fixing disconnects: Exploring the emergence of principled adaptations in a competency-based curriculum

PURPOSE

Competency-based medical education (CBME) promises to improve medical education through curricular reforms to support learner development. This intention may be at risk in the case of a Canadian approach to CBME called Competence by Design (CBD), since there have been negative impacts on residents. According to Joseph Schwab, teachers, learners and milieu must be included in the process of curriculum-making to prevent misalignments between intended values and practice. This study considered what can be learned from the process of designing, enacting and adapting CBD to better support learners.

METHODS

This qualitative study explored the making of CBD through the perspectives of implementation leads (N = 18) at national, institutional and programme levels. A sociomaterial orientation to agency in curriculum-making guided the inductive approach to interviewing and analysis in phase one. A deductive analysis in phase two applied Schwab's theory to further understand sources of misalignments and the purpose of adaptive responses.

RESULTS

Misalignments occurred when the needs of teachers, learners and milieu were initially underestimated in the process of curriculum-making, disconnecting assessment practices from experiences of teaching, learning and entrustment. While technical and structural issues posed significant constraints on agency, some implementation leads were able to make changes to the curriculum or context to fix the disconnects. We identified six purposes for principled adaptations to align with CBME values of responsive teaching, individualised learning and meaningful entrustment.

CONCLUSION

Collectively, the adaptations we characterise demonstrate constructive alignment, a foundational principle of CBME in which assessment and teaching work together to support learning. This study proposes a model for making context-shaped, values-based adaptations to CBME to achieve its promise.

Exploring the adoption of concept-based curricula: insights from educators and implications for change

Concept-based approaches to curriculum design have been proposed to solve content and curricula overload and promote conceptual learning. Few health professions have adopted this approach and little is known about how to support this educational change. We aimed to understand how nutrition and dietetics educators may navigate proposed education change towards concept-based curricula. We employed an interpretivist approach and in-depth interviews that explored the views of nutrition and dietetic educators towards using a concept-based approach to curriculum. Employing deductive thematic analysis based on the diffusion of innovation theory, data from twenty experienced dietetics educators were analysed. Three main themes were identified; the need for change champions, concerns about change, and the complexity of the education system. Diffusion of innovation theory highlighted that to enact change, the relative advantage and compatibility of the approach with current structures and systems, with evidence from trialling and observing the new approach in action, were needed. Developing education leaders and infiltrating the social system of education through existing communities of practice is critical to enacting educational change.

The Need for Competency-Based Education

OBJECTIVES

While pharmacy education updates learning as new information arises, changes to learning experiences can trail behind current practices and technology. There have been multiple calls for radical changes in how health professions education is delivered to ensure patients are receiving high-quality care. Competency-based education has been one way discussed in the literature for how to handle this need to develop students who have a willingness to learn and can problem-solve. The goal of this review is to examine whether competency-based education is needed to drive the profession of pharmacy forward.

FINDINGS

To address, we collaboratively identified stakeholder perspectives to evaluate the need. The following stakeholders achieved consensus among the committee members: patients/society, learners, workplace/profession, and academic institutions.

SUMMARY

Based on those perspectives, needs, and gaps to address those needs were identified and are presented in this review.

Unveiling the paradoxes of implementing post graduate competency based medical education programs

PURPOSE

Competency-based medical education (CBME) has gained prominence as an innovative model for post-graduate medical education, yet its implementation poses significant challenges, especially with regard to its sustainability. Drawing on paradox theory, we suggest that revealing the paradoxes underlying these challenges may contribute to our understanding of post graduate competency-based medical education (PGCBME) implementation processes and serve as a first-step in enhancing better implementation. Thus, the purpose of the current study is to identify the paradoxes associated with PGCBME implementation.

METHOD

A qualitative study was conducted, as part of a larger action research, using in-depth semi-structured interviews with fellows and educators in eight Neonatal wards.

RESULTS

Analysis revealed that the PGCBME program examined in this study involves three different levels of standardization, each serving as one side of paradoxical tensions; (1) a paradox between the need for standardized assessment tools and for free-flow flexible assessment tools, (2) a paradox between the need for a standardized implementation process across all wards and the need for unique implementation protocols in each ward; and 3) a paradox between the need for a standardized meaning of competency proficiency and the need for flexible and personal competency achievement indicators.

CONCLUSIONS

Implementing PGCBME programs involves many challenges, some of which are paradoxical, i.e. two contradictory challenges in which solving one challenge exacerbates another. Revealing these paradoxes is important in navigating them successfully.

Evaluating the Strengths and Barriers of Competency-Based Education in the Health Professions

OBJECTIVE

This study aimed to define competency-based education (CBE) for pharmacy education and describe how strengths and barriers of CBE can support or hinder implementation.

FINDINGS

Sixty-five studies were included from a variety of health professions in order to define competency based pharmacy education (CBPE) and identify barriers and benefits from the learner, faculty, institution, and society perspectives. From the 7 identified thematic categories, a CBPE definition was developed: "Competency-based pharmacy education is an outcomes-based curricular model of an organized framework of competencies (knowledge, skills, attitudes) for pharmacists to meet health care and societal needs. This learner-centered curricular model aligns authentic teaching and learning strategies and assessment (emphasizing workplace assessment and quality feedback) while deemphasizing time."

SUMMARY

This article provides a definition of CBE for its application within pharmacy education. The strengths and barriers for CBE were elucidated from other health professions' education literature. Identified implementation strengths and barriers aid in the discussions on what will support or hinder the implementation of CBE in pharmacy education.

The Pregnant Arthroplasty Surgeon: A Women in Arthroplasty Committee Editorial

BACKGROUND

Women orthopaedic surgeons face unique challenges during their careers. There are extremely low numbers of women in the field, particularly in the specialty of adult reconstruction. Factors contributing to low numbers of women entering this subspecialty include increased perceived physical demand relative to other fields, occupational hazards during pregnancy such as exposure to radiation and polymethylmethacrylate bone cement, concerns for work-life balance, and limited number of women within the subspecialty. The following editorial provides a framework to understand and manage the potential occupational hazards to pregnant and lactating surgeons, parental leave, and postpartum return to work. We aim to dispel any unfounded myths and provide evidence-based education that may help overcome these barriers. In doing so, we hope to encourage more women to consider adult reconstruction as a potential career.

METHODS

Our primary method consisted of completing an extensive literature review on the past and current articles about the aforementioned barriers which may contribute to the low number of women entering adult reconstruction. After this literature search was completed, we composed a comprehensive editorial that provided evidence-based education and recommendations for medical professionals.

CONCLUSIONS

Issues pertaining to parenthood, pregnancy, and lactation pose barriers to success for women in orthopedic surgery. These concerns may dissuade talented women from pursuing a rewarding career in adult reconstruction. Education on these issues is needed to help our early-career colleagues plan and care for their families. Clearly stated and published policies should be made available in all training programs, fellowships, and clinical practices to allow understanding and unbiased implementation. By being more inclusive, adult reconstruction will have access to the best possible surgeons, which will benefit not only patients but the field as a whole.

Enabling Implementation of Competency Based Medical Education through an Outcomes-Focused Accreditation System

Competency based medical education is being adopted around the world. Accreditation plays a vital role as an enabler in the adoption and implementation of competency based medical education, but little has been published about how the design of an accreditation system facilitates this transformation. The Canadian postgraduate medical education environment has recently transitioned to an outcomes-based accreditation system in parallel with the adoption of competency based medical education. Using the Canadian example, we characterize four features of an accreditation system that can facilitate the implementation of competency based medical education: theoretical underpinning, quality focus, accreditation standards, and accreditation processes. Alignment of the underlying educational theories within the accreditation system and educational paradigm drives change in a consistent and desired direction. An accreditation system that prioritizes quality improvement over quality assurance promotes educational system development and progressive change. Accreditation standards that achieve the difficult balance of being sufficiently detailed yet flexible foster a high fidelity of implementation without stifling innovation. Finally, accreditation processes that recognize the change process, encourage program development, and are not overly punitive all enable the implementation of competency based medical education. We also discuss the ways in which accreditation can simultaneously hinder the implementation of this approach. As education bodies adopt competency based medical education, particular attention should be paid to the role that accreditation plays in successful implementation.

Evaluating Competence by Design as a Large System Change Initiative: Readiness, Fidelity, and Outcomes

Program evaluation is an essential, but often neglected, activity in any transformational educational change. Competence by Design was a large-scale change initiative to implement a competency-based time-variable educational system in Canadian postgraduate medical education. A program evaluation strategy was an integral part of the build and implementation plan for CBD from the beginning, providing insights into implementation progress, challenges, unexpected outcomes, and impact. The Competence by Design program evaluation strategy was built upon a logic model and three pillars of evaluation: readiness to implement, fidelity and integrity of implementation, and outcomes of implementation. The program evaluation strategy harvested from both internally driven studies and those performed by partners and invested others. A dashboard for the program evaluation strategy was created to transparently display a real-time view of Competence by Design implementation and facilitate continuous adaptation and improvement. The findings of the program evaluation for Competence by Design drove changes to all aspects of the Competence by Design implementation, aided engagement of partners, supported change management, and deepened our understanding of the journey required for transformational educational change in a complex national postgraduate medical education system. The program evaluation strategy for Competence by Design provides a framework for program evaluation for any large-scale change in health professions education.

Guidelines and Recommendations for Targeted Neonatal Echocardiography and Cardiac Point-of-Care Ultrasound in the Neonatal Intensive Care Unit: An Update from the American Society of Echocardiography

Targeted neonatal echocardiography (TNE) involves the use of comprehensive echocardiography to appraise cardiovascular physiology and neonatal hemodynamics to enhance diagnostic and therapeutic precision in the neonatal intensive care unit. Since the last publication of guidelines for TNE in 2011, the field has matured through the development of formalized neonatal hemodynamics fellowships, clinical programs, and the expansion of scientific knowledge to further enhance clinical care. The most common indications for TNE include adjudication of hemodynamic significance of a patent ductus arteriosus, evaluation of acute and chronic pulmonary hypertension, evaluation of right and left ventricular systolic and/or diastolic function, and screening for pericardial effusions and/or malpositioned central catheters. Neonatal cardiac point-of-care ultrasound (cPOCUS) is a limited cardiovascular evaluation which may include line tip evaluation, identification of pericardial effusion and differentiation of hypovolemia from severe impairment in myocardial contractility in the hemodynamically unstable neonate. This document is the product of an American Society of Echocardiography task force composed of representatives from neonatology-hemodynamics, pediatric cardiology, pediatric cardiac sonography, and neonatology-cPOCUS. This document provides (1) guidance on the purpose and rationale for both TNE and cPOCUS, (2) an overview of the components of a standard TNE and cPOCUS evaluation, (3) disease and/or clinical scenario-based indications for TNE, (4) training and competency-based evaluative requirements for both TNE and cPOCUS, and (5) components of quality assurance. The writing group would like to acknowledge the contributions of Dr. Regan Giesinger who sadly passed during the final revisions phase of these guidelines. Her contributions to the field of neonatal hemodynamics were immense.

Exploring residents' perceptions of competency-based medical education across Canada: A national survey study

Background: As competency-based medical education (CBME) is implemented across Canada, little is known about residents' perceptions of this model. This study examined how Canadian residents understand CBME and their lived experiences with implementation. Methods: We administered a survey in 2018 with Likert-type and open-ended questions to 375 residents across Canada, of whom 270 were from traditional programs ("pre-CBME") and 105 were in a CBME program. We used the Mann-Whitney test to examine differences across samples, and analyzed qualitative data thematically. Results: Three themes were identified across both groups: program outcome concerns, changes, and emotional responses. In relation to program concerns, both groups were concerned about the administrative burden, challenges with the assessment process, and feedback quality. Only pre-CBME residents were concerned about faculty engagement and buy-in. In terms of changes, both groups discussed a more formalized assessment process with mixed reactions. Residents in the pre-CBME sample reported greater concerns for faculty time constraints, assessment completion, and quality of learning experiences, whilst those in CBME programs reported being more proactive in their learning and greater selfreflection. Residents expressed strong emotional narrative responses including greater stress and frustration in a CBME environment. Conclusion: Findings demonstrate that residents have mixed feelings and experiences regarding CBME. Their positive experiences align with the aim of developing more self-directed learners. However, the concerns suggest the need to address specific shortcomings to increase buy-in, while the emotional responses associated with CBME may require a cultural shift within residency programs to guard against burnout.

Evaluating Feedback Comments in Entrustable Professional Activities: A Cross-Sectional Study

INTRODUCTION

Competency-based medical education (CBME) has transformed postgraduate medical training, prioritizing competency acquisition over traditional time-based curricula. Integral to CBME are Entrustable Professional Activities (EPAs), that aim to provide high-quality feedback for trainee development. Despite its importance, the quality of feedback within EPAs remains underexplored.

METHODS

We employed a cross-sectional study to explore feedback quality within EPAs, and to examine factors influencing length of written comments and their relationship to quality. We collected and analyzed 1163 written feedback comments using the Quality of Assessment for Learning (QuAL) score. The QuAL aims to evaluate written feedback from low-stakes workplace assessments, based on 3 quality criteria (evidence, suggestion, connection). Afterwards, we performed correlation and regression analyses to examine factors influencing feedback length and quality.

RESULTS

EPAs facilitated high-quality written feedback, with a significant proportion of comments meeting quality criteria. Task-oriented and actionable feedback was prevalent, enhancing value of low-stakes workplace assessments. From the statistical analyses, the type of assessment tool significantly influenced feedback length and quality, implicating that direct and video observations can yield superior feedback in comparison to case-based discussions. However, no correlation between entrustment scores and feedback quality was found, suggesting potential discrepancies between the feedback and the score on the entrustability scale.

CONCLUSION

This study indicates the role of the EPAs to foster high-quality feedback within CBME. It also highlights the multifaceted feedback dynamics, suggesting the influence of factors such as feedback length and assessment tool on feedback quality. Future research should further explore contextual factors for enhancing medical education practices.

Knowledge and Perceptions of Competency-Based Medical Education in Diagnostic Radiology Post-Graduate Medical Education: Identifying Priorities and Developing a Framework for Professional Development Activities

Introduction: We evaluated knowledge and perceptions of an established Competency-Based Medical Education (CBME) model developed by the Royal College of Physicians and Surgeons of Canada, Competence by Design (CBD), and identified evidence-informed priorities for professional development activities (PDAs). Materials and Methods: Teaching faculty and residents at a single, large diagnostic radiology post-graduate medical education (PGME) program were eligible to participate in this cross-sectional, survey-based study. Knowledge of CBD was evaluated through multiple choice questions (MCQs), which assessed participants' understanding of major principles and terms associated with CBD. Participants' perceptions of the anticipated impact of CBD on resident education and patient care were evaluated and priorities for PDAs were identified, which informed a framework for CBD PDAs. Results: Fifty faculty and residents participated. The faculty and resident response rates were 11.6% (n = 29/249) and 55.3% (n = 21/38), respectively. The mean ± standard deviation overall score on MCQs was 39.0% ± 20.4%. The majority of participants perceived the impact of CBD on resident education to be equivocal and to not impact patient care. Knowledge of CBD was not statistically significantly associated with participants' perceptions of the impact of CBD on either resident education or patient care (P > .05). Delivery of high-quality feedback was the greatest priority identified for PDAs. Discussion: Our results and proposed CBD PDAs framework may help to guide diagnostic radiology PGME programs in designing evidence-informed PDAs, which may meaningfully contribute to the successful implementation of CBD in diagnostic radiology PGME. As diagnostic radiology PGME programs throughout the world increasingly implement CBME models, evidence-informed PDAs will become of increasing importance.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

Fitness-for-purpose of the CanMEDS competencies for workplace-based assessment in General Practitioner's Training: a Delphi study

BACKGROUND

In view of the exponential use of the CanMEDS framework along with the lack of rigorous evidence about its applicability in workplace-based medical trainings, further exploring is necessary before accepting the framework as accurate and reliable competency outcomes for postgraduate medical trainings. Therefore, this study investigated whether the CanMEDS key competencies could be used, first, as outcome measures for assessing trainees' competence in the workplace, and second, as consistent outcome measures across different training settings and phases in a postgraduate General Practitioner's (GP) Training.

METHODS

In a three-round web-based Delphi study, a panel of experts (n = 25-43) was asked to rate on a 5-point Likert scale whether the CanMEDS key competencies were feasible for workplace-based assessment, and whether they could be consistently assessed across different training settings and phases. Comments on each CanMEDS key competency were encouraged. Descriptive statistics of the ratings were calculated, while content analysis was used to analyse panellists' comments.

RESULTS

Out of twenty-seven CanMEDS key competencies, consensus was not reached on six competencies for feasibility of assessment in the workplace, and on eleven for consistency of assessment across training settings and phases. Regarding feasibility, three out of four key competencies under the role "Leader", one out of two competencies under the role "Health Advocate", one out of four competencies under the role "Scholar", and one out of four competencies under the role "Professional" were deemed as not feasible for assessment in a workplace setting. Regarding consistency, consensus was not achieved for one out of five competencies under "Medical Expert", two out of five competencies under "Communicator",one out of three competencies under "Collaborator", one out of two under "Health Advocate", one out of four competencies under "Scholar", one out of four competencies under "Professional". No competency under the role "Leader" was deemed to be consistently assessed across training settings and phases.

CONCLUSIONS

The findings indicate a mismatch between the initial intent of the CanMEDS framework and its applicability in the context of workplace-based assessment. Although the CanMEDS framework could offer starting points, further contextualization of the framework is required before implementing in workplace-based postgraduate medical trainings.

Defining new roles and competencies for administrative staff and faculty in the age of competency-based medical education

PURPOSE

These authors sought to define the new roles and competencies required of administrative staff and faculty in the age of CBME.

METHOD

A modified Delphi process was used to define the new CBME roles and competencies needed by faculty and administrative staff. We invited international experts in CBME (volunteers from the ICBME Collaborative email list), as well as faculty members and trainees identified via social media to help us determine the new competencies required of faculty and administrative staff in the CBME era.

RESULTS

Thirteen new roles were identified. The faculty-specific roles were: National Leader/Facilitator in CBME; Institutional/University lead for CBME; Assessment Process & Systems Designer; Local CBME Leads; CBME-specific Faculty Developers or Trainers; Competence Committee Chair; Competence Committee Faculty Member; Faculty Academic Coach/Advisor or Support Person; Frontline Assessor; Frontline Coach. The staff-specific roles were: Information Technology Lead; CBME Analytics/Data Support; Competence Committee Administrative Assistant.

CONCLUSIONS

The authors present a new set of faculty and staff roles that are relevant to the CBME context. While some of these new roles may be incorporated into existing roles, it may be prudent to examine how best to ensure that all of them are supported within all CBME contexts in some manner.

Competency-Based Medical Education (CBME): an Overview and Relevance to the Education of Future Surgical Oncologists

In the next two decades, the global cancer burden is expected to rise by 47%, and the demand for global cancer surgery will increase by 52%. At present, only 25% of the estimated 80% of patients needing surgical intervention have access to timely surgical care. The shortage of a trained workforce of surgical oncologists is one of the main barriers to providing the optimal surgical intervention needed for cancer patients. Some of the contributing factors to the shortage of trained surgical oncologists are variations in the current global educational platforms, long training programs, and physician burnout. Therefore, the availability of a credible training framework and a sustainable certification pipeline for future surgical oncologists is critical to meet the global demand for an adequate healthcare workforce. The current surgical oncology educational program is a time-based construct that trains surgeons to function seamlessly in the multidisciplinary care of cancer patients. However, there is a lack of flexibility in the training framework and timeline despite differences in trainees' abilities. Developing a globally acceptable standard curriculum for surgical oncology training based on the competency-based medical education (CBME) framework and tailoring it to local needs can increase the surgical oncology workforce ready to tackle the rising cancer burden. However, successful implementation of the global CBME-based surgical oncology training curriculum requires an innovative approach to ensure that this framework produces a competent surgical oncologist that meets the local needs.

Multicentre implementation of a nursing competency framework at a provincial scale: A qualitative description of facilitators and barriers

RATIONALE

Nurses are responsible for engaging in continuing professional development throughout their careers. This implies that they use tools such as competency frameworks to assess their level of development, identify their learning needs, and plan actions to achieve their learning goals. Although multiple competency frameworks and guidelines for their development have been proposed, the literature on their implementation in clinical settings is sparser. If the complexity of practice creates a need for context-sensitive competency frameworks, their implementation may also be subject to various facilitators and barriers.

AIMS AND OBJECTIVES

To document the facilitators and barriers to implementing a nursing competency framework on a provincial scale.

METHODS

This multicentre study was part of a provincial project to implement a nursing competency framework in Quebec, Canada, using a three-step process based on evidence from implementation science. Nurses' participation consisted in the self-assessment of their competencies using the framework. For this qualitative descriptive study, 58 stakeholders from 12 organizations involved in the first wave of implementation participated in group interviews to discuss their experience with the implementation process and their perceptions of facilitators and barriers. Data were subjected to thematic analysis.

RESULTS

Analysis of the data yielded five themes: finding the 'right unit' despite an unfavourable context; taking and protecting time for self-assessment; creating value around competency assessment; bringing the project as close to the nurses as possible; making the framework accessible.

CONCLUSION

This study was one of the first to document the large-scale, multi-site implementation of a nursing competency framework in clinical settings. This project represented a unique challenge because it involved two crucial changes: adopting a competency-based approach focused on educational outcomes and accountability to the public and valorizing a learning culture where nurses become active stakeholders in their continuing professional development.

Optimal training in communication model (OPTiCOM): A programmatic roadmap

OBJECTIVES

Teaching primary care residents patient communication skills is essential, complex, and impeded by barriers. We find no models guiding faculty how to train residents in the workplace that integrate necessary system components, the science of physician-patient communication training and competency-based medical education. The aim of this project is to create such a model.

METHODS

We created OPTiCOM using four steps: (1) communication educator interviews, analysis and theme development; (2) initial model construction; (3) model refinement using expert feedback; (4) structured literature review to validate, refine and finalize the model.

RESULTS

Our model contains ten interdependent building blocks organized into four developmental tiers. The Foundational value tier has one building block Naming relationship as a core value. The Expertize and resources tier includes four building blocks addressing: Curricular expertize, Curricular content, Leadership, and Time. The four building blocks in the Application and development tier are Observation form, Faculty development, Technology, and Formative assessment. The Language and culture tier identifies the final building block, Culture promoting continuous improvement in teaching communication.

CONCLUSIONS

OPTiCOM organizes ten interdependent systems building blocks to maximize and sustain resident learning of communication skills. Practice Implications Residency faculty can use OPTiCOM for self-assessment, program creation and revision.

Making sense of competency-based medical education (CBME) literary conversations: A BEME scoping review: BEME Guide No. 78

BACKGROUND

Competency-based medical education (CBME) received increased attention in the early 2000s by educators, clinicians, and policy makers as a way to address concerns about physician preparedness and patient safety in a rapidly changing healthcare environment. Opinions and perspectives around this shift in medical education vary and, to date, a systematic search and synthesis of the literature has yet to be undertaken. The aim of this scoping review is to present a comprehensive map of the literary conversations surrounding CBME.

METHODS

Twelve different databases were searched from database inception up until 29 April 2020. Literary conversations were extracted into the following categories: perceived advantages, perceived disadvantages, challenges/uncertainties/skepticism, and recommendations related to CBME.

RESULTS

Of the 5757 identified records, 387 were included in this review. Through thematic analysis, eight themes were identified in the literary conversations about CBME: credibility, application, community influence, learner impact, assessment, educational developments, organizational structures, and societal impacts of CBME. Content analysis supported the development of a heat map that provides a visual illustration of the frequency of these literary conversations over time.

CONCLUSIONS

This review serves two purposes for the medical education research community. First, this review acts as a comprehensive historical record of the shifting perceptions of CBME as the construct was introduced and adopted by many groups in the medical education global community over time. Second, this review consolidates the many literary conversations about CBME that followed the initial proposal for this approach. These findings can facilitate understanding of CBME for multiple audiences both within and outside of the medical education research community.

Review of Challenges to the Implementation of Competence by Design in Post-Graduate Medical Education: What Can Diagnostic Radiology Learn from the Experience of Other Specialty Disciplines?

Competence by Design (CBD) is a medical education initiative instituted by the Royal College of Physicians and Surgeons of Canada to improve the training of resident physicians in specialty disciplines. CBD integrates Competency Based Medical Education with traditional specialty discipline post-graduate medical education (PGME) training through the application of an organizational framework of competencies. Various specialty disciplines in Canada have transitioned to CBD since 2017 in a staggered approach. Diagnostic radiology PGME programs in Canada are expected to transition to CBD in 2022 for the incoming resident physician cohort. This article reviews potential challenges to the implementation of CBD in diagnostic radiology PGME programs and proposes evidence-informed targeted strategies and solutions to address these challenges. It is important for diagnostic radiology PGME programs to understand the challenges pertaining to the implementation of CBD so that they may be able to successfully implement this or similar medical education initiatives in their programs. Moreover, as radiology subspecialty PGME programs, such as nuclear medicine, interventional radiology, neuroradiology, and pediatric radiology, likewise transition to CBD and diagnostic radiology PGME programs internationally increasingly implement other Competency Based Medical Education models, the implications of the challenges pertaining to the implementation of CBD will further become of increasing importance.

'Next steps are…': An exploration of coaching and feedback language in EPA assessment comments

PURPOSE

Entrustable Professional Activities (EPA) assessments are intended to facilitate meaningful, low-stakes coaching and feedback, partly through the provision of written comments. We sought to explore EPA assessment comments provided to internal medicine (IM) residents for evidence of feedback and coaching language as well as politeness.

METHODS

We collected all written comments from EPA assessments of communication from a first-year IM resident cohort at the University of Toronto. Sensitized by politeness theory, we analyzed data using principles of constructivist grounded theory.

RESULTS

Nearly all EPA assessments (94%) contained written feedback based on focused clinical encounters. The majority of comments demonstrated coaching language, including phrases like 'don't forget to,' and 'next steps are,' followed by specific suggestions for improvement. A variety of words, including 'autonomy' and 'independence' denoted entrustment decisions. Linguistic politeness strategies such as hedging were pervasive, seemingly to minimize harm to the supervisor-trainee relationship.

CONCLUSION

Evidence of written coaching feedback suggests that EPA assessment comments are being used as intended as a means of formative feedback to promote learning. Yet, the frequent use of polite language suggests that EPAs may be higher-stakes than expected, highlighting a need for changes to the assessment culture and improved feedback literacy.

Teaching Practical Skills in Anesthesia, Intensive Care, Emergency and Pain Medicine—What Is Really Relevant for Medical Students? Results of a German National Survey of Nearly 3000 Anesthesiologists

As a part of a major reform of the medical curriculum in Germany, the national catalogue of learning objectives is being revised with the focus shifting from theory-based learning to teaching practical skills. Therefore, we conducted an online survey to answer the question, which practical skills are essential in anesthesia. Participants were asked to rate the relevance of several skills, that medical students should be able to perform at the time of graduation. A total of 2898 questionnaires could be evaluated. The highest ratings were made for “bringing a patient into lateral recumbent position” and “diagnosing a cardiac arrest”. All learning objectives regarding regional anesthesia were rated as irrelevant. Furthermore, learning objectives like “performing a bronchoscopy” or “performing a rapid sequence induction” had low ratings. In the subgroup analysis, physicians with advanced training and those who were working at university hospitals rated most skills with higher relevance compared to others. Our survey provides a good prioritization of practical skills for the development of new curricula and assessment frameworks. The results can also help to establish our discipline as a cross-sectional subject in competency-based medical education, thus further increasing the attractiveness for medical students.

Changing Dynamics of Accreditation in Oral and Maxillofacial Surgery

The Commission on Dental Accreditation is responsible for accrediting postgraduate dental education programs, including residencies and fellowships using a well-delineated process that includes an annual survey of various aspects of the program related to the Standards for Advanced Dental Education Programs in Oral and Maxillofacial Surgery and a program self-study and site visit with a review by the Specialty Education Review Committee of the Commission and the Commission on a 5-year cycle. The primary goal of the accreditation process is to assure the public, the profession, and the institution that the program is in compliance with the published standards.

Ready, set, go! Evaluating readiness to implement competency-based medical education

PURPOSE

Organizational readiness is critical for successful implementation of an innovation. We evaluated program readiness to implement Competence by Design (CBD), a model of Competency-Based Medical Education (CBME), among Canadian postgraduate training programs.

METHODS

A survey of program directors was distributed 1 month prior to CBD implementation in 2019. Questions were informed by the R = MC² framework of organizational readiness and addressed: program motivation, general capacity for change, and innovation-specific capacity. An overall readiness score was calculated. An ANOVA was conducted to compare overall readiness between disciplines.

RESULTS

Survey response rate was 42% (n = 79). The mean overall readiness score was 74% (30-98%). There was no difference in scores between disciplines. The majority of respondents agreed that successful implementation of CBD was a priority (74%), and that their leadership (94%) and faculty and residents (87%) were supportive of change. Fewer perceived that CBD was a move in the right direction (58%) and that implementation was a manageable change (53%). Curriculum mapping, competence committees and programmatic assessment activities were completed by >90% of programs, while <50% had engaged off-service disciplines.

CONCLUSION

Our study highlights important areas where programs excelled in their preparation for CBD, as well as common challenges that serve as targets for future intervention to improve program readiness for CBD implementation.

The Importance of Sociocultural Factors in Moderating the Applicability of Test-Enhanced Learning to Continuing Professional Development

Many studies have shown that repeated testing promotes learning, an effect known as test-enhanced learning. Although test-enhanced learning has been well-established within health care trainees, there are mixed findings of whether testing facilitates learning within continuing professional development (CPD) contexts. This piece argues that for testing to support learning in CPD, there is need to understand the various social, cultural, and structural factors that influence clinicians' willingness to participate in regular testing activities. The solution to this problem is conceptual in nature, whereby CPD programs must consider how learning cultures shape practicing clinicians' perspectives toward testing.

Implementation of Competency-Based Medical Education in a Canadian Medical Oncology Training Program: a First Year Retrospective Review

As part of a university-wide initiative, competency-based medical education (CBME) was implemented in the Medical Oncology training program at Queen's University in July 2017. Stages, entrustable professional activities (EPAs), and required training experiences established by the Royal College of Physicians and Surgeons of Canada (RCPSC) national subspecialty committee were adopted. Entrada (Elentra), the electronic portfolio developed at Queen's University, was used for assessment collection. Between July 2017 and December 2018, participating faculty members completed 157 assessments. Eighty-nine percent were EPA assessments with a median of 16 assessments per faculty member (range 1-40). Ninety-five percent of assessments included written "Comments" or "Next steps" with 56% of all assessments including specific or actionable feedback. Discussions between the program director, residents, program administrator, CBME education consultant, and CBME lead led to the identification of 9 lessons learned during implementation. These centered on (1) faculty and resident development and engagement; (2) sharing the work of CBME; (3) collaboration and communication; (4) global assessment; (5) assessment plan challenges; (6) burden of CBME; (7) limitations of e-portfolio; (8) importance of early tracking of resident progress; and (9) culture change. This article describes the experience of the authors and considers strategies that may be helpful to programs implementing CBME in their teaching and learning environment.

The Senior Medical Resident's New Role in Assessment in Internal Medicine

PURPOSE

With the introduction of competency-based medical education, senior residents have taken on a new, formalized role of completing assessments of their junior colleagues. However, no prior studies have explored the role of near-peer assessment within the context of entrustable professional activities (EPAs) and competency-based medical education. This study explored internal medicine residents' perceptions of near-peer feedback and assessment in the context of EPAs.

METHOD

Semistructured interviews were conducted from September 2019 to March 2020 with 16 internal medicine residents (8 first-year residents and 8 second- and third-year residents) at the University of Toronto, Toronto, Ontario, Canada. Interviews were conducted and coded iteratively within a constructivist grounded theory approach until sufficiency was reached.

RESULTS

Senior residents noted a tension in their dual roles of coach and assessor when completing EPAs. Senior residents managed the relationship with junior residents to not upset the learner and potentially harm the team dynamic, leading to the documentation of often inflated EPA ratings. Junior residents found senior residents to be credible providers of feedback; however, they were reticent to find senior residents credible as assessors.

CONCLUSIONS

Although EPAs have formalized moments of feedback, senior residents struggled to include constructive feedback comments, all while knowing the assessment decisions may inform the overall summative decision of their peers. As a result, EPA ratings were often inflated. The utility of having senior residents serve as assessors needs to be reexamined because there is concern that this new role has taken away the benefits of having a senior resident act solely as a coach.

Students' perceptions on newly designed foundation course at MBBS entry level: A mixed-method study

BACKGROUND

Selection of students to medical colleges is based on merit of candidate at written entrance examination which does not take care of nonscholastic abilities of the students. Medicine is a profession that requires not only mastery of large body of knowledge and clinical skills but also high standards of professionalism and appropriate attitudes.

METHODOLOGY

A 1-month long foundation course was conducted for the fresh students admitted to MBBS course at Andaman and Nicobar Islands Institute of Medical Sciences, Port Blair, India. Seventy-four students had joined the course. The course was planned, arranged, and executed by the department of Medical Education Unit. It included six modules. Retrospective pre- and postevaluation was done for knowledge, skill, and attitude. Wilcoxon signed-rank test was used for all pair-wise comparisons. Participants also completed a questionnaire (Likert scale and some open-ended items) about educational quality of sessions and utility of the course. Descriptive statistics and qualitative methods were used for data analysis.

RESULTS AND CONCLUSION

Response rate was 100%. Students perceived foundation course as an excellent opportunity to acquire basic knowledge, attitude, and skills required for subsequent phases in MBBS course. This Foundation Course enhanced their orientation to the curriculum and campus, boosted confidence to cope up with the challenging syllabus, helped them develop time and stress management skills, and also sensitized them to achieve the highest standards of morale required to ensure ethical practice.

Learner Phenotypes in Competency-Based Medical Education

With the launch of competency-based medical education internationally, the practical realities of implementation have failed to live up to many of the proposed theoretical benefits. Specifically, as educators we have observed a number of assessment challenges that seem directly related to identified learner phenotypes. This commentary seeks to describe these specific learner phenotypes, along with actionable recommendations for programs and their competence committees in order to overcome the associated obstacles in assessment. We describe strategies related to both the individual and program level, which can be utilized for both short-term adjustments and long-term programmatic transformation.

A tale of two frameworks: charting a path to lifelong learning for physician leaders through CanMEDS and LEADS

PURPOSE

The purpose of this paper was to determine the complementarity between the Canadian Medical Education Directions for Specialists (CanMEDS) physician competency and LEADS leadership capability frameworks from three perspectives: epistemological, philosophical and pragmatic. Based on those findings, the authors propose how the frameworks collectively layout pathways of lifelong learning for physician leadership.

DESIGN/METHODOLOGY/APPROACH

Using a qualitative approach combining critical discourse analysis with a modified Delphi, the authors examined "How complementary the CanMEDS and LEADS frameworks are in guiding physician leadership development and practice" with the following sub-questions: What are the similarities and differences between CanMEDS and LEADS from: An epistemological and philosophical perspective? The perspective of guiding physician leadership training and practice? How can CanMEDS and LEADS guide physician leadership development from medical school to retirement?

FINDINGS

Similarities and differences exist between the two frameworks from philosophical and epistemological perspectives with significant complementarity. Both frameworks are founded on a caring ethos and value physician leadership - CanMEDS (for physicians) and LEADS (physicians as one of many professions) define leadership similarly. The frameworks share beliefs in the function of leadership, embrace a belief in distributed leadership, and although having some philosophical differences, have a shared purpose (preparing for changing health systems). Practically, the frameworks are mutually supportive, addressing leadership action in different contexts and where there is overlap, complement one another in intent and purpose.

ORIGINALITY/VALUE

To the best of the authors' knowledge, this is the first paper to map the CanMEDS (physician competency) and LEADS (leadership capabilities) frameworks. By determining the complementarity between the two, synergies can be used to influence physician leadership capacity needed for today and the future.

Development of a veterinary emergency open standard competency framework using a competency-based model of medical education

OBJECTIVE

To address the shortage of emergency veterinarians, the profession is exploring accelerated training pathways. We sought to contribute to the solution by developing the foundation for an open standard, competency-based veterinary emergency training curriculum for use by any program. We also developed a curricular delivery, tracking, and assessment system to demonstrate how the framework can be integrated into training programs.  DESIGN: Hybrid Delphi method.  SETTING: Academia and referral practice.  ANIMALS: None.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

An emergency veterinary competency framework was developed by adapting the human Model of the Clinical Practice of Emergency Medicine, which aligns with the Competency-Based Veterinary Education framework, to produce 4 areas of core competency: Patient Care, Interpersonal/Communication, Professionalism, and Practice-based Learning/Improvement. A comprehensive list of veterinary emergency skills was generated and organized within the framework utilizing the hybrid Delphi method. An initial survey completed by 133 emergency and critical care specialists and emergency room clinicians produced data regarding the value of specific skills. An 11-member focus group consisting of survey participants iterated upon the survey results to produce a master library of skills and cases, including 56 Patient Care, 43 Interpersonal/Communication, 11 Practice-based Learning/Improvement, and 20 Professionalism skills, as well as 155 case types. The curricular delivery system tracks and assesses case management proficiency and development of knowledge and professional skills using a patient care eLearning program and simulation training environment.  CONCLUSIONS: The increasing need for emergency veterinarians is a shared industry-wide challenge. To contribute toward a collective solution, we have undergone an evidence-based process to create the foundation for an open standard competency framework composed of a library of skills and cases. We offer this open standard framework to the veterinary profession and hope it continues to grow and evolve as we drive toward developing competency-based training programs that address the shortage of emergency veterinarians.

Acceptance and validity of the methods used to implement a competency based medical education programme in an Intensive Care Department of a teaching referral center

OBJECTIVES

1. To determine the satisfaction of tutors and residents with a specific methodology used to implement CoBaTrICE. 2. To determine the reliability and validity of the global rating scales designed ad hoc to assess the performance of the residents for training purposes.

DESIGN

Prospective cohort study.

PARTICIPANTS

All the residents and tutors of the ICU Department of the Hospital Universitario y Politécnico la Fe de Valencia.

INTERVENTION

CoBaTrICE implementation started in March 2016, it was based on: (1) Training the tutors in feedback techniques; (2) Performing multiple objective and structured work based assessments to achieve the competences of the program; and (3) The use of an electronic portfolio to promote learning reflection and to collect the evidence that learning was taking place.

METHODS

The acceptance of CoBaTrICE was explored through a satisfaction survey conducted after 9 months of implementation of the training program. The 15 residents and 5 tutors of the ICU Department were asked about the methodology of the formative assessments, the quality of the feedback, self-learning regulation and the electronic portfolio usefulness. The validity of the global rating scales was assessed through the tests alfa de Cronbach, reliability and generalizability indexes, and intraclass correlation coefficient.

RESULTS

The implementation of CoBaTrICE was satisfactory in all the dimensions studied. The global rating scales used for formative purposes showed reliability and validity.

CONCLUSIONS

The methodology used to implement CoBaTrICE was highly valued by tutors and residents. The global rating scales used for formative purposes showed reliability and validity.

Using the Five Core Components of Competency-Based Medical Education to Support Implementation of CBVE

In recent years, veterinary education has begun the transition to competency-based models, recognizing that, like medical education, our goals include improved patient and client outcomes and the importance of learner-centered methods in education. Given that implementation of competency-based veterinary education (CBVE) is still in its relative infancy across many veterinary programs, we stand to gain from a unified approach to its implementation. As a guideline, the five core components of competency-based medical education (CBME) should serve to ensure and maintain fidelity of the original design of outcomes-based education during implementation of CBVE. Identified the essential and indispensable elements of CBME which include 1) clearly articulated outcome competencies required for practice, 2) sequenced progression of competencies and their developmental markers, 3) tailored learning experiences that facilitate the acquisition of competencies, 4) competency-focused instruction that promotes the acquisition of competencies, and 5) programmatic assessment. This review advocates the adoption of the principles contained in the five core components of CBME, outlines the approach to implementation of CBVE based upon the five core components, and addresses the key differences between veterinary and medical education which may serve as challenges to ensuring fidelity of CBVE during implementation.

PJplus - a project improving practical training during the final year of medical education

BACKGROUND

Practical training on the patient is crucial in medical students' last year education - the so-called practical year (PJ) in Germany. Due to difficulties in combining student training with the everyday tasks on ward, it is often criticised as not sufficient for a good preparation for later practical work. The Medical Faculty of the University of Jena therefore designed a project called "PJplus". The project includes mentoring and workplace-based assessment by means of Mini-Clinical Evaluation Exercise (Mini-CEX) in combination with training workshops for supervisors. Three years after the first clinical departments started voluntary participation, the project was evaluated by comparing the experience and the self-assessed learning progress of students from departments participating in this project (PJplus group) with those non-participating (control group).

METHODS

An online questionnaire was sent to all medical students registered at the University of Jena for PJ between March 2016 and April 2017. The students were invited to participate at the end of each section of their PJ within that period. The answers of the PJplus group were compared to the answers of the control group using descriptive and multivariable analysis.

RESULTS

201 students participated in the survey and filled out 257 questionnaires. PJplus was recommended by 80% of the students participating in the project. The PJplus group (n=92) was significantly more satisfied with their PJ and felt significantly better prepared for work than the control group (n=165). The project's elements mentoring and feedback could lead to a better improvement in practical medical skills. However, only 17% of the students managed to conduct the required amount of three Mini-CEX during their PJ rotation and 52% of the students seemed to have lost contact to their mentor or did not have one at all. These difficulties arose due to unfamiliarity with the project, shortage of time and staff on ward or due to lack of motivation among supervisors.

CONCLUSION

Adding mentoring and feedback to the PJ helps to better prepare students for their practical work after finishing studies. With the project presented, it is feasible to integrate these elements in a structured way. Nevertheless, a good control of the elements' implementation and consistent training of the supervising physicians is needed to ensure long-term success.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

Simulation Interventions for the Classroom to Support the Acquisition of Interprofessional Competencies

Interprofessional collaboration (IPC) supports the delivery of quality and safe healthcare. The acquisition of interprofessional competencies both pre-licensure and post-licensure are key to implementing this approach in the healthcare workplace. This report documents the development and implementation of a simulation intervention to support interprofessional education (IPE) in the undergraduate classroom for pre-licensure learners. The learning activity framework includes an exposure phase (e.g., didactic classroom instruction) and an immersion phase (e.g., simulation intervention and debriefing). Details on the debriefing process are included as it is key to achieving the learning objectives. The three learning activity pilot tests (n=150) revealed that learners recognized that interprofessional competencies were an important asset to support IPC. The pilot tests identified the need for further development in order for students to make a connection with the mastery phase (i.e., clinical placement). The next steps will include the development and incorporation of formative tools to assess learners' progress, as well as a plan to evaluate the learning activity that will connect all three phases (exposure, immersion, and mastery) of the learning framework.

El estado de la investigación en educación en cirugía general en Colombia (2000-2020): un análisis bibliométrico

Introducción. Se desconoce el estado de la investigación en educación médica en cirugía general (IEMC) en Colombia en el siglo XXI. El objetivo de esta revisión bibliométrica de la literatura es realizar un análisis de las publicaciones relacionadas con la IEMC en Colombia a partir del año 2000. Métodos. Se incluyeron artículos originales, escritos en español o inglés, publicados por grupos de investigación que cuentan al menos con un investigador colombiano como autor principal o coautor. Se definieron variables demográficas e indicadores bibliométricos para cada estudio y autor. Resultados. Un total de 63 estudios fueron incluidos. Estas publicaciones se enfocaron predominantemente en investigación sobre bienestar, enseñanza clínica y simulación a nivel de postgrado. El 36 % de los artículos sobre IEMC fueron publicados en revistas sin indexación (ISI/SCOPUS); 13 artículos (20,6 %) fueron publicados en revistas en el cuartil 1 (Q1). El promedio de citas por artículo fue 9,3. Discusión. La producción intelectual en educación en cirugía en Colombia tiene bajo impacto a nivel internacional. Los hallazgos encontrados pueden ser utilizados para organizar y priorizar la investigación en educación quirúrgica en el país.

Lessons from the implementation of developmental progress assessment: A scoping review

OBJECTIVES

Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts.

METHODS

We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results.

RESULTS

Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences.

CONCLUSIONS

Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.

Seven ways to get a grip on implementing Competency-Based Medical Education at the program level

Competency-based medical education (CBME) curricula are becoming increasingly common in graduate medical education. Put simply, CBME is focused on educational outcomes, is independent of methods and time, and is composed of achievable competencies.¹ In spite of widespread uptake, there remains much to learn about implementing CBME at the program level. Leveraging the collective experience of program leaders at Queen's University, where CBME simultaneously launched across 29 specialty programs in 2017, this paper leverages change management theory to provide a short summary of how program leaders can navigate the successful preparation, launch, and initial implementation of CBME within their residency programs.

Process and outcome evaluation of a CBME intervention guided by program theory

RATIONALE

Competency-based medical education (CBME) has gained momentum as an improved training model, but literature on outcomes of CBME, including evaluation of implementation processes, is minimal. We present a case for the following: (a) the development of a program theory is essential prior to or in the initial stages of implementation of CBME; (b) the program theory should guide the strategies and methods for evaluation that will answer questions about anticipated and unintended outcomes; and (c) the iterative process of testing assumptions and hypotheses will lead to modifications to the program theory to inform best practices of implementing CBME.

METHODS

We use the Triple C Competency-based Curriculum as a worked example to illustrate how process and outcome evaluation, guided by a program theory, can lead to meaningful enhancement of CBME curriculum, assessment, and implementation strategies. Using a mixed methods design, the processes and outcomes of Triple C were explored through surveys, interviews, and historical document review, which captured the experiences of various stakeholders.

FINDINGS

The theory-led program evaluation process was able to identify areas that supported CBME implementation: the value of a strong nondirective national vertical core supporting the transformation in education, program autonomy, and adaptability to pre-existing local context. Areas in need of improvement included the need for ongoing support from College of Family Physicians of Canada (CFPC) and better planning for shifts in program leadership over time.

CONCLUSIONS

Deliberately pairing evaluation alongside change is an important activity and, when accomplished, yields valuable information from the experiences of those implementing and experiencing a program. Evaluation and the development of an updated program theory facilitate the introduction of new changes and theories that build on these findings, which also supports the desired goal of contributing toward cumulative science rather than "reinventing the wheel."

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

Construct: This study seeks to determine validity evidence for the Quality of Assessment for Learning score (QuAL score), which was created to evaluate short qualitative comments that are related to specific scores entered into a workplace-based assessment, common within the competency-based medical education (CBME) context. Background: In the age of CBME, qualitative comments play an important role in clarifying the quantitative scores rendered by observers at the bedside. Currently there are few practical tools that evaluate mixed data (e.g. associated score-and-comment data), other than the comprehensive Completed Clinical Evaluation Report Rating tool (CCERR) that was originally derived to rate end-of-rotation reports. Approach: A multi-center, randomized cohort-based rating exercise was conducted to evaluate the rating properties of the QuAL score as compared to the CCERR. One group rated comments using the QuAL score, and the other group rated comments using the CCERR. A generalizability study (G-Study) and a decision study (D-study) were conducted to determine the number of meta-raters for a reliable rating (phi-coefficient target of >0.80). Both scores were correlated against rater's gestalt perceptions of utility for both faculty and residents reading the scores. Results: Twenty-five meta-raters from 20 sites participated in this rating exercise. The G-study revealed that the CCERR group (n = 13) rated the comments with a very high reliability (Phi = 0.97). Meanwhile, the QuAL group (n = 12) rated the comments with a similarly high reliability (Phi = 0.97). The QuAL score required only two raters to reach an acceptable target reliability of >0.80, while the CCERR required three. The QuAL score correlated with perceptions of utility (Meta-rater usefulness, Pearson's r = 0.69, p < 0.001; Perceived usefulness for trainee, r = 0.74, p < 0.001). The CCERR performed similarly, correlating with perceived faculty (r = 0.67, <0.001) and resident utility (0.79, <0.001). Conclusions: The QuAL score is reliable rating score that correlates well with perceptions of utility. The QuAL score may be useful for rating shorter comments generated by workplace-based assessments.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation

PURPOSE

Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change.

METHOD

Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences.

RESULTS

In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow.

CONCLUSIONS

Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Approaching culture in medical education: Three perspectives

BACKGROUND

The notion of culture is increasingly invoked in the medical education literature as a key influence on how educational strategies unfold, and culture change is frequently identified as a necessary precursor to progress. A meaningful perspective on what culture means is often missing from these discussions, however. Without a theoretically grounded notion of culture, calls for culture change are challenging to interpret and to act upon.

OBJECTIVE

In this cross-cutting edge paper, we explore how culture has been defined and theorised using three lenses: the organisational perspective; the identity perspective, and the practice perspective. We consider what each perspective might offer to medical education researchers.

RESULTS

Each of these perspectives draws on a range of disciplinary influences, and none represents a singular theory of culture. Broadly, the organisational perspective directs our attention to the shared assumptions and values that bind individuals within an organisation. It tends to view culture through a strategic lens; culture may be either a barrier to or a facilitator of the changes that are inevitably required of an organisation if it is to maintain its relevance. The identity perspective, particularly the notion of figured worlds, alerts us to the power of communal narratives to shape how individuals see themselves within particular cultural worlds. The practice perspective emphasises what actually occurs in practice, avoiding symbolic ideas about culture and shared values and instead privileging activity and human-material networks or arrangements.

CONCLUSIONS

These diverse perspectives share a common thread- they shift our research gaze beyond the individual, allowing us instead to see how those individuals form organisations, inhabit cultural worlds and constitute practices. They afford substance and direction for explorations of culture, and thus offer the promise of a more nuanced understanding of some of medical education's most challenging problems.

Developing a National Competency-Based Diabetes Curriculum in Undergraduate Medical Education: A Delphi Study

OBJECTIVES

In recent years there has been an increased emphasis on competency-based medical education (CBME) in Canada and internationally, as can be seen with the implementation of competency-based curriculums for postgraduate medical education (PGME) through the Royal College of Physicians and Surgeons of Canada. Currently, no Canada-wide consensus exists on educational competencies relating to diabetes in undergraduate medical education (UGME). Our aim in this study was to develop a list of competencies and objectives for UGME in diabetes using a modified Delphi method.

METHODS

Representatives involved in the development of the diabetes curriculum at all 17 medical schools across Canada were contacted. A draft list of competencies and objectives was developed by the research team using the existing curriculums at 9 Canadian medical schools and was organized using the CanMEDS framework. A Delphi method was used, with 2 iterations in order to reach consensus.

RESULTS

Twelve of 17 medical schools agreed to participate. Of the 12 surveys sent in the first round, 8 responses were received (response rate 66.7%). The revised version was then resent to the 8 respondents and 7 responses were received (response rate 87.5%). A list of 9 competencies and 62 objectives was finalized.

CONCLUSIONS

A competency-based consensus curriculum for diabetes education for undergraduate medical students was developed using a modified Delphi method. The final consensus syllabus will be disseminated across the country. This curriculum serves as a step in the transition to competency-based UGME and in ensuring that future medical school graduates are proficient in diabetes care.