What resources do clinical competency committees (CCCs) require to do their work? A pilot study comparing CCCs across specialties

Clinical competency committee perceptions of entrustable professional activities and their value in assessing fellows: A qualitative study of pediatric subspecialty program directors

OBJECTIVES

To examine the composition and processes of Clinical Competency Committees (CCCs) assigning entrustable professional activity (EPA) levels of supervision for pediatric subspecialty fellows and to examine fellowship program director (FPD) perspectives about using EPAs to determine fellows' graduation readiness.

METHODS

A qualitative study was performed using one-on-one interviews with a purposeful sample of pediatric subspecialty FPDs to yield a thematic analysis. Semi-structured interview guides were used for participants who self-identified as EPA users or non-users. Inductive analysis and coding were performed on transcripts until theoretical sufficiency was attained.

RESULTS

Twenty-eight FPDs were interviewed. There was significant variability in the composition and processes of CCCs across subspecialties. FPDs felt that CCCs intuitively understand what entrustment means, allowing for ease of application of level of supervision (LOS) scales and consensus. FPDs perceived that EPAs provided a global assessment of fellows and are one tool to determine graduation readiness.

CONCLUSIONS

Although there was variability in the makeup and processes of CCCs across subspecialties, FPDs believe EPAs are intuitive and relatively easy to implement. Consensus can be reached easily using EPA-specific LOS scales focusing on entrustment. FPDs desire a better understanding of how EPAs should be used for graduation.

Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

OBJECTIVE

Program directors in surgical disciplines need more tools from the ACGME to help them use Milestone ratings to improve trainees' performance. This is especially true in competencies that are notoriously difficult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS competency categories.

DESIGN

This retrospective cohort study utilized Milestone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended graduation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of trainees not reaching the recommended graduation target based on their previous Milestone ratings.

SETTING

All ACGME-accredited vascular surgery training programs within the United States.

PARTICIPANTS

All trainees completing a 2 year vascular surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses.

RESULTS

The overall rate of failing to achieve the recommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all trainees. For trainees with a Milestone rating at ≤ 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended graduation target ranged from 37.0% to 71.5% across sub-competencies, with the highest risks observed under PROF for "Administrative Tasks" (71.5%) and under ICS for "Communication with the Healthcare Team" (56.7%).

CONCLUSIONS

As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Milestone assessment ratings of PROF and ICS during early training can be used to predict achievement of competency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.

A multispecialty ethnographic study of clinical competency committees (CCCs)

PURPOSE

Clinical competency committees (CCCs) assess residents' performance on their specialty specific milestones, however there is no 'one-size fits all' blueprint for accomplishing this. Thus, CCCs have had to develop their own procedures. The goal of this study was to examine these efforts to assist new programs embarking on this venture and established programs looking to improve their CCC practices and processes.

METHODS

We purposefully sampled CCCs across multiple specialties and institutions. Data from three sources were triangulated: (1) online demographic survey, (2) ethnographic observations of CCC meetings and (3) post-observation semi-structured interviews with the program director and/or CCC chairperson. Template analysis was used to build the coding structure.

RESULTS

Sixteen observations were completed with 15 different CCCs at 9 institutions. Three main thematic categories that impact the operations of CCCs emerged: (1) Membership structure and members roles, (2) Roles of the CCC in residency and 3) CCC processes, including trainee presentation to the committee and decision-making. While effective practices were observed, substantial variation existed in all three thematic areas.

CONCLUSIONS

While CCCs used some known effective practices, substantial variation in structure and processes was notable across CCCs. Future work should explore the impact of this variation on educational outcomes.

Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation

PURPOSE

This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME).

METHODS

A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design.

RESULTS

Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions.

CONCLUSIONS

Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.

Competence committees: The steep climb from concept to implementation

INTRODUCTION

Competence committees (CCs) are groups of educators tasked with reviewing resident progress throughout their training, making decisions regarding the achievement of Entrustable Professional Activities and recommendations regarding promotion and remediation. CCs have been mandated as part of competency-based medical education programmes worldwide; however, there has yet to be a thorough examination of the implementation challenges they face and how this impacts their functioning and decision-making processes. This study examined CC implementation at a Canadian institution, documenting the shared and unique challenges that CCs faced and overcame over a 3-year period.

METHODS

This study consisted of three phases, which were conceptually and analytically linked using Moran-Ellis and colleagues' notion of 'following a thread.' Phase 1 examined the early perceptions and experiences of 30 key informants using a survey and semi-structured interviews. Phase 2 provided insight into CCs' operations through a survey sent to 35 CC chairs 1-year post-implementation. Phase 3 invited 20 CC members to participate in semi-structured interviews to follow up on initial themes 2 years post-implementation. Detailed observation notes from 16 CC meetings across nine disciplines were used to corroborate the findings from each phase.

RESULTS

Response rates in each phase were 83% (n = 25), 43% (n = 15) and 60% (n = 12), respectively. Despite the high degree of support for CCs among faculty and resident members, several ongoing challenges were highlighted: adapting to programme size, optimising membership, engaging residents, maintaining capacity among members, sharing and aggregating data and developing a clear mandate.

DISCUSSION

Findings of this study reinforce the importance of resident engagement and information sharing between disciplines. Challenges faced by CCs are discussed in relation to the existing literature to inform a better understanding of group decision-making processes in medical education. Future research could compare implementation practices across sites and explore which adaptations lead to better or worse decision-making outcomes.