Faculty development in the age of competency-based medical education: A needs assessment of Canadian emergency medicine faculty and senior trainees

Unveiling the paradoxes of implementing post graduate competency based medical education programs

PURPOSE

Competency-based medical education (CBME) has gained prominence as an innovative model for post-graduate medical education, yet its implementation poses significant challenges, especially with regard to its sustainability. Drawing on paradox theory, we suggest that revealing the paradoxes underlying these challenges may contribute to our understanding of post graduate competency-based medical education (PGCBME) implementation processes and serve as a first-step in enhancing better implementation. Thus, the purpose of the current study is to identify the paradoxes associated with PGCBME implementation.

METHOD

A qualitative study was conducted, as part of a larger action research, using in-depth semi-structured interviews with fellows and educators in eight Neonatal wards.

RESULTS

Analysis revealed that the PGCBME program examined in this study involves three different levels of standardization, each serving as one side of paradoxical tensions; (1) a paradox between the need for standardized assessment tools and for free-flow flexible assessment tools, (2) a paradox between the need for a standardized implementation process across all wards and the need for unique implementation protocols in each ward; and 3) a paradox between the need for a standardized meaning of competency proficiency and the need for flexible and personal competency achievement indicators.

CONCLUSIONS

Implementing PGCBME programs involves many challenges, some of which are paradoxical, i.e. two contradictory challenges in which solving one challenge exacerbates another. Revealing these paradoxes is important in navigating them successfully.

Knowledge and Perceptions of Competency-Based Medical Education in Diagnostic Radiology Post-Graduate Medical Education: Identifying Priorities and Developing a Framework for Professional Development Activities

Introduction: We evaluated knowledge and perceptions of an established Competency-Based Medical Education (CBME) model developed by the Royal College of Physicians and Surgeons of Canada, Competence by Design (CBD), and identified evidence-informed priorities for professional development activities (PDAs). Materials and Methods: Teaching faculty and residents at a single, large diagnostic radiology post-graduate medical education (PGME) program were eligible to participate in this cross-sectional, survey-based study. Knowledge of CBD was evaluated through multiple choice questions (MCQs), which assessed participants' understanding of major principles and terms associated with CBD. Participants' perceptions of the anticipated impact of CBD on resident education and patient care were evaluated and priorities for PDAs were identified, which informed a framework for CBD PDAs. Results: Fifty faculty and residents participated. The faculty and resident response rates were 11.6% (n = 29/249) and 55.3% (n = 21/38), respectively. The mean ± standard deviation overall score on MCQs was 39.0% ± 20.4%. The majority of participants perceived the impact of CBD on resident education to be equivocal and to not impact patient care. Knowledge of CBD was not statistically significantly associated with participants' perceptions of the impact of CBD on either resident education or patient care (P > .05). Delivery of high-quality feedback was the greatest priority identified for PDAs. Discussion: Our results and proposed CBD PDAs framework may help to guide diagnostic radiology PGME programs in designing evidence-informed PDAs, which may meaningfully contribute to the successful implementation of CBD in diagnostic radiology PGME. As diagnostic radiology PGME programs throughout the world increasingly implement CBME models, evidence-informed PDAs will become of increasing importance.

Defining new roles and competencies for administrative staff and faculty in the age of competency-based medical education

PURPOSE

These authors sought to define the new roles and competencies required of administrative staff and faculty in the age of CBME.

METHOD

A modified Delphi process was used to define the new CBME roles and competencies needed by faculty and administrative staff. We invited international experts in CBME (volunteers from the ICBME Collaborative email list), as well as faculty members and trainees identified via social media to help us determine the new competencies required of faculty and administrative staff in the CBME era.

RESULTS

Thirteen new roles were identified. The faculty-specific roles were: National Leader/Facilitator in CBME; Institutional/University lead for CBME; Assessment Process & Systems Designer; Local CBME Leads; CBME-specific Faculty Developers or Trainers; Competence Committee Chair; Competence Committee Faculty Member; Faculty Academic Coach/Advisor or Support Person; Frontline Assessor; Frontline Coach. The staff-specific roles were: Information Technology Lead; CBME Analytics/Data Support; Competence Committee Administrative Assistant.

CONCLUSIONS

The authors present a new set of faculty and staff roles that are relevant to the CBME context. While some of these new roles may be incorporated into existing roles, it may be prudent to examine how best to ensure that all of them are supported within all CBME contexts in some manner.

Postgraduate Surgical Education in East, Central, and Southern Africa: A Needs Assessment Survey

BACKGROUND

The Lancet Commission on Global Surgery has identified workforce development as an important component of National Surgical Plans to advance the treatment of surgical disease in low- and middle-income countries. The goal of our study is to identify priorities of surgeon educators in the region so that collaboration and intervention may be appropriately targeted.

STUDY DESIGN

The American College of Surgeons Operation Giving Back, in collaboration with leaders of the College of Surgeons of Eastern, Central and Southern Africa (COSECSA), developed a survey to assess the needs and limitations of surgical educators working under their organizational purview. COSECSA members were invited to complete an online survey to identify and prioritize factors within 5 domains: (1) Curriculum Development, (2) Faculty Development, (3) Structured Educational Content, (4) Skills and Simulation Training, and (5) Trainee Assessment and Feedback.

RESULTS

One-hundred sixty-six responses were received after 3 calls for participation, representing all countries in which COSECSA operates. The majority of respondents (78%) work in tertiary referral centers. Areas of greatest perceived need were identified in the Faculty Development and Skills and Simulation domains. Although responses differed between domains, clinical responsibilities, cost, and technical support were commonly cited as barriers to development.

CONCLUSIONS

This needs assessment identified educational needs and priorities of COSECSA surgeons. Our study will serve as a foundation for interventions aimed at further improving graduate surgical education and ultimately patient care in the region.

A Needs Assessment Survey of Division Chiefs at an Academic Children's Hospital

Purpose

The Division Chief at an academic health sciences centre has many leadership roles and responsibilities. There are no data on leadership training needs for Division Chiefs, and so we sought to design and implement a needs assessment for pediatric Division Chiefs at CHEO, a pediatric academic health sciences centre in Eastern Ontario, Canada.

Methods

A needs assessment survey was developed with the aim to document demographics, preparedness for the role of Division Chief and desired leadership training for the role. This survey was piloted, revised and then distributed to all the Division Chiefs at our institution. The results of each question were collated, and simple descriptive statistics were calculated.

Results

The survey was completed by 22 of 31 Division Chiefs. The majority of respondents were from the Department of Pediatrics (63.6%), followed by Surgery (20%), Psychiatry (3.3%) and Laboratory Medicine (3.3%). Their mean length of time as Division Chief was 5.5 years. Seventy-seven percent had concurrent leadership roles in addition to the role of Division Chief. None felt they were very well prepared for the role, five felt they were somewhat well prepared, nine were neutral, five were somewhat unprepared and three were very unprepared for the role. Half of the respondents received mentoring, either formal or informal, for their role and all but one felt that formal mentoring would have been useful. In terms of desired training, the Division Chiefs felt they had the most knowledge and skills in patient safety. All wanted training in developing divisional budgets, and many desired training in supporting the academic mission of the Division.

Conclusion

Overall, this needs assessment identified an unmet need for leadership training and development among Division Chiefs. The findings are being used to optimize onboarding of Division Chiefs and an ongoing leadership development program targeted at this group.

Overcoming the barriers to implementation of competence-based medical education in post-graduate medical education: a narrative literature review

To ensure that residents are equipped with the necessary skills for practice, competence-based medical education (CBME) represents a transformative change in postgraduate medical education, which is being progressively introduced across Canadian specialty residency programs. Successful implementation will require adjustments to curriculum, assessment, and evaluation, with careful attention to the unique needs in the local context, including resident and faculty development. This narrative review of the literature aimed to determine the potential barriers to the successful implementation of CBME and the strategies by which they can be addressed, with a specific consideration of the author's program in pediatrics in Manitoba. Eleven articles were identified with a specific focus on the implementation of CBME in the post-graduate setting, and 10 were included in the review after critical appraisal. Three key themes emerged from the articles: the value of broad stakeholder engagement and leadership, the importance of faculty and resident development, and the development of specific support systems for the educational curriculum. Different strategies were considered and contrasted for addressing these important themes. This review provides important insights and practical approaches to the barriers that should be useful as programs prepare for the implementation of CBME.

Review of Challenges to the Implementation of Competence by Design in Post-Graduate Medical Education: What Can Diagnostic Radiology Learn from the Experience of Other Specialty Disciplines?

Competence by Design (CBD) is a medical education initiative instituted by the Royal College of Physicians and Surgeons of Canada to improve the training of resident physicians in specialty disciplines. CBD integrates Competency Based Medical Education with traditional specialty discipline post-graduate medical education (PGME) training through the application of an organizational framework of competencies. Various specialty disciplines in Canada have transitioned to CBD since 2017 in a staggered approach. Diagnostic radiology PGME programs in Canada are expected to transition to CBD in 2022 for the incoming resident physician cohort. This article reviews potential challenges to the implementation of CBD in diagnostic radiology PGME programs and proposes evidence-informed targeted strategies and solutions to address these challenges. It is important for diagnostic radiology PGME programs to understand the challenges pertaining to the implementation of CBD so that they may be able to successfully implement this or similar medical education initiatives in their programs. Moreover, as radiology subspecialty PGME programs, such as nuclear medicine, interventional radiology, neuroradiology, and pediatric radiology, likewise transition to CBD and diagnostic radiology PGME programs internationally increasingly implement other Competency Based Medical Education models, the implications of the challenges pertaining to the implementation of CBD will further become of increasing importance.

Preparedness of Residents and Medical Students for the Transition to Competence by Design in Diagnostic Radiology Post-Graduate Medical Education

Introduction: This needs assessment evaluated residents' and medical students' knowledge of Competence by Design (CBD), perceived benefits of and challenges or barriers to the transition to CBD for residents, and perceived overall preparedness for the transition to CBD in diagnostic radiology. Materials and Methods: All diagnostic radiology residents and medical students in Canada were eligible to participate in this national cross-sectional, questionnaire-based needs assessment. Knowledge of CBD was evaluated through participants' self-reported rating of their knowledge of CBD on a 5-point Likert scale. Perceived benefits of and challenges or barriers to the transition to CBD for residents were rank ordered. Participants' overall self-reported preparedness for the transition to CBD was assessed on a 5-point Likert scale. Data were summarized by descriptive statistics and bivariate analyses were conducted as appropriate. Results: Ninety-four residents (n = 77) and medical students (n = 17) participated in this needs assessment. Participants' mean ± standard deviation self-reported rating of their overall knowledge of CBD was 2.86 ± .94. Provision of meaningful feedback to learners and learners' ability to identify their own educational needs were among the highest ranked perceived benefits of the transition to CBD, while demands on time and increased frequency of evaluation were among the highest ranked perceived challenges or barriers to the transition to CBD. Few participants reported being either "prepared" (4.7%) or "somewhat prepared" (14.0%) for the transition to CBD. Conclusion: Preparedness for the transition to CBD in diagnostic radiology may be improved. Targeted interventions to augment the preparedness of residents and medical students should be considered.

The Impact of Electronic Data to Capture Qualitative Comments in a Competency-Based Assessment System

Introduction

Digitalizing workplace-based assessments (WBA) holds the potential for facilitating feedback and performance review, wherein we can easily record, store, and analyze data in real time. When digitizing assessment systems, however, it is unclear what is gained and lost in the message as a result of the change in medium. This study evaluates the quality of comments generated in paper vs. electronic media and the influence of an assessor’s seniority.

Methods

Using a realist evaluation framework, a retrospective database review was conducted with paper-based and electronic medium comments. A sample of assessments was examined to determine any influence of the medium on the word count and the Quality of Assessment for Learning (QuAL) score. A correlation analysis evaluated the relationship between word count and QuAL score. Separate univariate analyses of variance (ANOVAs) were used to examine the influence of the assessor's seniority and medium on word count, QuAL score, and WBA scores.

Results

The analysis included a total of 1,825 records. The average word count for the electronic comments (M=16) was significantly higher than the paper version (M=12; p=0.01). Longer comments positively correlated with QuAL score (r=0.2). Paper-based comments received lower QuAL scores (0.41) compared to electronic (0.51; p<0.01). Years in practice was negatively correlated with QuAL score (r=-0.08; p<0.001) as was word count (r=-0.2; p<0.001).

Conclusion

Digitization of WBAs increased the length of comments and did not appear to jeopardize the quality of WBAs; these results indicate higher-quality assessment data. True digital transformation may be possible by harnessing trainee data repositories and repurposing them to analyze for faculty-relevant metrics.

Redesigning continuing professional development: Harnessing design thinking to go from needs assessment to mandate

BACKGROUND

The world of medicine is constantly changing, and with it the continuing professional development (CPD) needs of physicians. As the CPD landscape is shifting away from unidirectional delivery of knowledge through live large group learning (conferences) and is placing increased emphasis on new approaches for skills training not taught a decade ago, a new approach is needed.

APPROACH

Using design thinking techniques, we hosted a full-day retreat for emergency medicine stakeholders in Hamilton and the surrounding region. Prior to the retreat we collected medico-legal data on emergency physicians in our region and performed a needs assessment survey. At the retreat, we had participants brainstorm ideas for CPD, generate archetypes for end-users, then generate solutions to the problems they had identified. These proposals were presented to the larger group for feedback and refinement.

EVALUATION

The Design Thinking Retreat generated five main pillars for action by our CPD team. 1) Simulation/procedural learning (staff simulation, procedural skills day, in situ simulation); 2) Asynchronous learning (website and podcast); 3) Synchronous learning (small group sessions for staff); 4) Community connectivity (online platform for collaboration and communication); and 5) Coaching & mentorship (focused coaching for specific practice improvement, improved onboarding for new staff).

REFLECTION

These ideas have vastly increased engagement in CPD. Stakeholder consultation via design thinking may be a key approach for educators to use.

Beyond the CLAIM: A comprehensive needs assessment strategy for creating an Advanced Medical Education Research Training Program (ARMED-MedEd)

BACKGROUND

The health professions education (HPE) landscape has shifted substantively with increasing professionalization of research and scholarship. Clinician educators seeking to become competitive in this domain often pursue fellowships and master's degrees in HPE, but there are few resources for the continuing professional development (CPD) of those who seek to continue developing their scholarly practice within HPE. Acknowledging the multiple players in this landscape, the authors sought to design a new "beyond beginners" HPE research program using a novel needs assessment planning process.

METHODS

The authors developed and conducted a new three-phase, five-step process that sets forth a programmatic approach to conducting a needs assessment for a CPD course in HPE research. The five steps of the CLAIM method are: Competitive analysis, Literature review with thematic analysis, Ask stakeholders, Internal review by experts, and Mapping of a curriculum. These steps are organized into three phases (Discovery, Convergence, and Synthesis).

RESULTS

Over a 12-month period, the authors completed a comprehensive needs assessment. The CLAIM process revealed that longitudinal digital connection, diverse and in depth exposure to HPE research methods, skills around scholarly publishing, and leadership and management of research would be beneficial to our design.

CONCLUSIONS

The CLAIM method provided scaffolding to help the authors create a robust curriculum that adopts a scholarly approach for developing a HPE research course. This needs assessment methodology may be useful in other CPD contexts.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

OBJECTIVES

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

METHODS

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

RESULTS

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

CONCLUSIONS

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Faculty-lead Opinions on Workplace-based Methods for Graduated Managerial Teaching (FLOW MGMT): A National Cross-sectional Survey of Canadian Emergency Medicine Lead Educators

OBJECTIVES

Patient volumes are increasing in emergency departments (ED), causing issues with long wait times and overcrowding. One strategy to cope with this phenomenon is to focus on improving patient flow through the ED. Building on earlier work that identified how staff physicians manage flow and what techniques they employ to teach managerial skills to residents, we aimed to determine when it was most appropriate to implement these teaching strategies in a resident's training.

METHODS

We employed a Canada-wide cross-sectional survey of experienced emergency medicine (EM) teaching faculty to determine when they felt our previously identified teaching strategies would be appropriate to implement. The survey was piloted with local educational experts.

RESULTS

A total of 21 EM (38% female, 62% male) educators from 11 programs responded to the survey. The respondents provided an average of 42.5 endorsements per participant for specific teaching techniques across the stages of training. The core of discipline (35.9%) and transition to practice (39.7%) were the stages of training that received the most endorsement. The top two teaching techniques included the observational teaching technique "attitudinal role modeling (i.e. a strong work ethic)" and the conversational teaching technique "teacher provides clinical pearls, tips, pointers." The participants showed fairly high agreement, with the advanced in situ techniques showing fairly high reliability as measured by intraclass correlation coefficients ranging from 0.88 to 0.90.

CONCLUSIONS

Our results show a trend toward faculty utilizing more didactic and observational teaching techniques early in residency and then progressing toward more experiential techniques in the senior stages of training. This is consistent with a graduated increase in responsibility as residents demonstrate competency and progress through their training. The results of this study will help inform faculty development around teaching managerial skills in the area of competency-based medical education.

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

BACKGROUND

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement.

OBJECTIVE

We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels.

METHODS

Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations.

RESULTS

Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested.

CONCLUSIONS

There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

Regional needs assessment for emergency physician audit and feedback

Objectives

Audit and feedback is widely used to improve physician performance. Many data metrics are being provided to physicians, yet most of these are driven by the regulatory environment. We sought to conduct a needs assessment of audit and feedback metrics that were most useful to clinicians within our health care region.

Methods

We conducted a Web-based survey of five clinical practice sites in our region and asked that physicians rank 49 clinical practice metrics. In addition, we assessed their readiness for audit and feedback and their preferences for data confidentiality. We collected data on duration of training, gender, and site of practice (academic v. community) allowing for comparison between groups.

Results

A total of 104 emergency medicine physicians participated in the survey (52.3% response rate). There was a significant readiness for participation in audit and feedback activities. Top ranked metrics were emergency department return rates and colleague's assessment of collegiality and quality of care, which were common across all sites. Small yet significant differences were noted between genders and academic v. community practitioners.

Conclusion

This study represents the first regional analysis of physician preferences for audit and feedback activities and implementation. It demonstrates that physicians are interested in audit and feedback activities and provides a roadmap for the development of a regional audit and feedback structure. It will also be used as a guiding document for regional change management.