Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment

Evaluating a medication history-taking entrustable professional activity and its assessment tool - Survey of a statewide public hospital pharmacy service

INTRODUCTION

Entrustable Professional Activities (EPAs) are tasks that professionals within a field perform autonomously. EPAs are incorporated in workplace-based assessment tools to assist training and professional development. Few studies have evaluated medication history-taking EPAs use in pharmacy practice and none have sought stakeholder feedback on their use. This study evaluates the quality of the medication history-taking EPA utilized in South Australian public hospitals and the usability of its assessment tool.

METHODS

A voluntary online questionnaire was conducted from July 15th to September 2nd 2021 to gather the opinions of stakeholders on the use of the medication history-taking EPA. The questionnaire was developed based on tools identified in the literature and utilized 14 open-text and five-point Likert scale questions. The questionnaire was distributed using Survey Monkey® to a purposive sample of staff and students.

RESULTS

82 responses were received from 218 surveys distributed, yielding a response rate of 38%. Respondents believed the EPA promotes learner development (90.6%) and the provision of useful feedback (83%). 94.3% considered the EPA to be easy to use but only 56.6% indicated that using it fits easily within their workday. Time constraints and the presence of context-specific descriptors were commonly perceived as limitations. Some stakeholders indicated a lack of understanding of entrustment decisions.

CONCLUSION

The EPA and its assessment tool were perceived to have good quality and usability. Reducing the length of the tool, broadening its applicability across contexts, and improving user understanding of entrustment decision-making may support better use of the tool.

Training and Sustaining: Training and Learning Collaborative Outcomes Across a Statewide Network for Early Autism Diagnosis

OBJECTIVE

The objective of this study was to describe the development of a primary care professional (PCP) autism diagnosis training model and to report on outcomes related to PCP training and sustained engagement in a longitudinal learning collaborative.

METHODS

We developed Accelerating the Diagnosis of Autism with Primary care Training (ADAPT), a training program to prepare PCPs to develop independent competency in evaluation of autism in children aged 14 to 48 months. ADAPT includes didactic and case-based modules and practice-based coaching delivered by an expert diagnostic specialist; after training, PCPs participate in a longitudinal learning collaborative. Aligned with competency-based medical education standards, measures of autism evaluation knowledge and diagnostic competency are collected.

RESULTS

From 2021 to 2023, 13 PCPs completed ADAPT didactic and practicum training to reach competency in independent autism evaluation. Clinicians demonstrated significant improvement in total autism knowledge after didactic training (p = 0.02). Scoring agreement on an autism observational assessment tool between clinicians and expert diagnosticians improved over case observations and practicum evaluations. Similarly, PCPs demonstrated improved evaluation competence, moving on average from Advanced Beginner to Competent Performer as rated by expert diagnosticians. After training, PCPs attended 57% of monthly learning collaborative sessions.

CONCLUSION

Training PCPs to deliver autism evaluations for young children as part of tiered community-based models of care is a promising solution to address access and waitlist challenges. ADAPT is an intensive, standardized PCP training model that results in achievement of independent competency and sustained engagement in autism evaluation. Effectiveness-implementation studies are needed to ensure scalability and sustainability of training models.

Faculty Entrustment and Resident Entrustability

Importance

As the surgical education paradigm transitions to entrustable professional activities, a better understanding of the factors associated with resident entrustability are needed. Previous work has demonstrated intraoperative faculty entrustment to be associated with resident entrustability. However, larger studies are needed to understand if this association is present across various surgical training programs.

Objective

To assess intraoperative faculty-resident behaviors and determine if faculty entrustment is associated with resident entrustability across 4 university-based surgical training programs.

Design, Setting, and Participants

This cross-sectional study was conducted at 4 university-based surgical training programs from October 2018 to May 2022. OpTrust, a validated tool designed to assess both intraoperative faculty entrustment and resident entrustability behaviors independently, was used to assess faculty-resident interactions. A total of 94 faculty and 129 residents were observed. Purposeful sampling was used to create variation in type of operation performed, case difficulty, faculty-resident pairings, faculty experience, and resident training level.

Main Outcomes and Measures

Observed resident entrustability scores (scale 1-4, with 4 indicating full entrustability) were compared with reported measures (faculty level, case difficulty, resident postgraduate year [PGY], resident gender, observation month) and observed faculty entrustment scores (scale 1-4, with 4 indicating full entrustment). Path analysis was used to explore direct and indirect effects of the predictors. Associations between resident entrustability and faculty entrustment scores were assessed by pairwise Pearson correlation coefficients.

Results

A total of 338 cases were observed. Cases observed were evenly distributed by faculty experience (1-5 years' experience: 67 [20.9%]; 6-14 years' experience: 186 [58%]; ≥15 years' experience: 67 [20.9%]), resident PGY (PGY 1: 28 [8%]; PGY 2: 74 [22%]; PGY 3: 64 [19%]; PGY 4: 40 [12%]; PGY 5: 97 [29%]; ≥PGY 6: 36 [11%]), and resident gender (female: 183 [54%]; male: 154 [46%]). At the univariate level, PGY (mean [SD] resident entrustability score range, 1.44 [0.46] for PGY 1 to 3.24 [0.65] for PGY 6; F = 38.92; P < .001) and faculty entrustment (2.55 [0.86]; R2 = 0.94; P < .001) were significantly associated with resident entrustablity. Path analysis demonstrated that faculty entrustment was associated with resident entrustability and that the association of PGY with resident entrustability was mediated by faculty entrustment at all 4 institutions.

Conclusions and Relevance

Faculty entrustment remained associated with resident entrustability across various surgical training programs. These findings suggest that efforts to develop faculty entrustment behaviors may enhance intraoperative teaching and resident progression by promoting resident entrustability.

Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

Use of learner-driven, formative, ad-hoc, prospective assessment of competence in physical therapist clinical education in the United States: a prospective cohort study

PURPOSE

The purpose of this project was to implement a process for learner-driven, formative, prospective, ad-hoc, entrustment assessment in Doctor of Physical Therapy clinical education. Our goals were to develop an innovative entrustment assessment tool, and then explore whether the tool detected (1) differences between learners at different stages of development and (2) differences within learners across the course of a clinical education experience. We also investigated whether there was a relationship between the number of assessments and change in performance.

METHODS

A prospective, observational, cohort of clinical instructors (CIs) was recruited to perform learner-driven, formative, ad-hoc, prospective, entrustment assessments. Two entrustable professional activities (EPAs) were used: (1) gather a history and perform an examination and (2) implement and modify the plan of care, as needed. CIs provided a rating on the entrustment scale and provided narrative support for their rating.

RESULTS

Forty-nine learners participated across 4 clinical experiences (CEs), resulting in 453 EPA learner-driven assessments. For both EPAs, statistically significant changes were detected both between learners at different stages of development and within learners across the course of a CE. Improvement within each CE was significantly related to the number of feedback opportunities.

CONCLUSION

The results of this pilot study provide preliminary support for the use of learner-driven, formative, ad-hoc assessments of competence based on EPAs with a novel entrustment scale. The number of formative assessments requested correlated with change on the EPA scale, suggesting that formative feedback may augment performance improvement.

Competency-Based Cultural Safety Training in Medical Education at La Sabana University, Colombia: A Roadmap of Curricular Modernization

Issue: Cultural safety enhances equitable communication between health care providers and cultural groups. Most documented cultural safety training initiatives focus on Indigenous populations from high-income countries, and nursing students, with little research activity reported from low- and middle-income countries. Several cultural safety training initiatives have been described, but a modern competency-based cultural safety curriculum is needed. Evidence: In this article, we present the Competency-Based Education and Entrustable Professional Activities frameworks of the Faculty of Medicine at La Sabana University in Colombia, and illustrate how this informed modernization of medical education. We describe our co-designed cultural safety training learning objectives and summarize how we explored its impact on medical education through mixed-methods research. Finally, we propose five cultural safety intended learning outcomes adapted to the updated curriculum, which is based on the Competency-Based Education model. Implications: This article presents five cultural safety intended learning outcomes for undergraduate medical education. These learning outcomes are based on Competency-Based Education and the Entrustable Professional Activities framework and can be used by faculties of medicine interested in including the cultural safety approach in their curriculum.

Examining the effects of early patient care and biomedical science integration on predoctoral dental student competence and confidence

INTRODUCTION

Our study investigates early experiential learning as a method of curricular integration by allowing students to begin their clinical experience in the first year of the programme, as well as distributing biomedical classes throughout the predoctoral dental school curriculum.

MATERIALS AND METHODS

This study utilises a quasi-experimental design with two different groups, Standard Curriculum Group and Integrated Curriculum Group, n = 87. Data were collected from 2017 to 2021.

RESULTS

We found that, on average, it took 608 h less for the participants in an integrated curriculum group to reach clinical competence in comparison to peers who did not experience the same methods of integration in their programme. These data were collected through daily faculty evaluations of students' progression as well as participants' own self-assessment. Our results indicate that participants in the Integrated Curriculum Group also experienced a positive effect on their confidence in their ability to apply the biomedical sciences to patient care.

DISCUSSION/CONCLUSION

Our findings demonstrate that predoctoral dental programmes may be able to bring about positive outcomes for students' clinical confidence and competence by providing patient care opportunities early in the programme and sequencing the biomedical sciences throughout the curriculum. As such, it appears that early experiential learning may be a viable option for curricular integration that can have a positive effect on both students' confidence in their clinical abilities and their progression to clinical competence.

Competence committees decision-making; an interplay of data, group orientation, and intangible impressions

BACKGROUND

The implementation of competency-based medical education and utilization of competence committees (CC) represents a paradigm shift in residency education. This qualitative study aimed to explore the methods used by two operational CC and their members to make decisions about progression and competence of their residents.

METHODS

An instrumental case study methodology was used to study the CC of two postgraduate training programs. Transcripts from observed CC meetings, member interviews, and guiding documents were analyzed using a constructivist grounded theory approach to reveal themes explaining the decision-making process.

RESULTS

Our study found that the CC followed a process that began within a social decision schema model and evolved to a discussion that invoked social influence theory, shared mental models, and social judgment scheme to clarify the points of contention. We identified that the CC decision-making was at risk of bias, primarily influenced by the group composition, the group orientation and individual members' mindset, as well as their personal experiences with the trainees.

CONCLUSIONS

Increased awareness of the sources of bias in CC functioning and familiarity with the CC role in competency-based medical education would enable committees to provide valuable feedback to all trainees regardless of their trajectory.

Assessing the utility of a novel entrustment-supervision assessment tool

BACKGROUND

Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools.

METHODS

The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor.

RESULTS

A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8.

CONCLUSION

The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.

Academic Faculty Demonstrate Weak Agreement in Evaluating Orthopaedic Surgery Residents

Background

As objective metrics fade, subjective elements of orthopaedic surgery applicants carry increasing importance during recruitment. Academic orthopaedic surgeons believe they can select for high-performing orthopaedic residents. However, can they agree? The purpose of this study was (1) to analyze an academic orthopaedic surgery department to determine whether they agree on which residents performed best during residency and; (2) to correlate preresidency and intraresidency factors with postresidency evaluations of resident performance.

Methods

With Institutional Review Board [IRB] approval, an orthopaedic surgery department completed surveys to evaluate residency performance for 10 years of graduated residents (2012-2022). Faculty determined (1) Post-Residency Class Rank (PRCR)-ranked from the highest (1) to lowest performing resident (5) relative to their class based on faculty perspective of performance-and (2) Rank List Score (RLS)-ranked based off the 5-point AOA SLOR used during recruitment interviews. RLS assessed how likely the department would have graduates in the program again based on residency performance. Free marginal Cohen's kappa statistics assessed faculty inter-rater agreement. Preresidency metrics (United States Medical Licensing Exam [USMLE] 1 and 2 scores, research publications, etc) were correlated with Orthopaedic In-Training Exam (OITE) scores, research productivity, American Board of Orthopaedic Surgery (ABOS) scores, and faculty-derived rankings. Linear regressions with forward variable entry (p < 0.05) were used to determine factors associated with excellent resident performance.

Results

Eighteen of 25 faculty members (72%) provided survey responses evaluating 46 residents. Faculty agreed 37% and 38% of the time for PRCR (kappa 0.26) and RLS (kappa 0.23), respectively. Step 2 score was the only preresidency factor significantly associated with PRCR (p = 0.03, r2 = 0.15) and RLS (p = 0.02, r2 = 0.3). The only intraresidency factor significantly correlated with PRCR (p = 0.002, r2 = 0.50) and RLS (p = 0.01, r2 = 0.39) was PGY-4 OITE score.

Conclusions

An academic orthopaedic surgery department is able to come to a consensus on evaluations of residency performance relative to peers in the same year of training (PRCR) and an objective standard (RLS). Step 2 and Post-Graduate Year (PGY)-4 OITE scores were the only preresidency and intraresidency factors with significant association to higher postresidency, faculty-derived performance scores.

Level of Evidence

III.

Establishing entrustable professional activities for psychiatry residents in China

PURPOSE

The authors established entrustable professional activities for psychiatry residents in China.

METHODS

The authors conducted a literature research and two expert consultation rounds following the Delphi method in 2022 to screen and optimize entrustable professional activities for psychiatry residents.

RESULTS

The effective questionnaire recovery rate in the two consultation rounds was 100% (44/44). The expert authority coefficients of the first and second consultation rounds were 0.861 and 0.881, respectively. The Kendall harmony coefficients of the first and second expert consultation rounds were 0.279 (χ2 = 405.43, P < .001) and 0.389 (χ2 = 3456.83, P < .001), respectively. The arithmetic means of the various indicators' evaluation results in the two consultation rounds ranged between 3.61 and 4.93, and the full score rates were between 13.6% and 93.2%. The authors established 17 entrustable professional activities for psychiatry residents and their contents with phase-based modularization and formulated the entrustable level of each at various stages.

CONCLUSIONS

Combined with standardized psychiatry training characteristics, the authors preliminarily established phase-specific and modular entrustable professional activities for psychiatry residents. The formulated entrustable professional activities are suitable for the practice and clinical environment of standardized psychiatry training in China. The devised system has good observability and measurability and provides a simple and feasible competency evaluation method for standardized psychiatry resident training.

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs

Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.

Residents as supervisors: How senior residents make ad hoc entrustment decisions

BACKGROUND

As residency programmes transition to competency-based medical education, there has been substantial inquiry into understanding how ad hoc entrustment decisions are made by attending supervisors in the clinical context. However, although attendings are ultimately responsible for the decisions and actions of resident trainees, senior residents are often the ones directly supervising junior residents enrolled in competency-based training programmes. This clinical dynamic has been largely overlooked in the ad hoc entrustment literature. The purpose of this study was to explore the considerations senior residents entertain when making ad hoc entrustment decisions for their junior resident colleagues.

METHODS

In semi-structured interviews, 11 senior resident supervisors (third, fourth and fifth year) in obstetrics and gynaecology described how they entrust junior residents with clinical activities in the moment. Following constructivist grounded theory methodology, data were iteratively collected and coded with constant comparison until theoretical sufficiency was determined.

RESULTS

Senior residents described many similar considerations as attendings regarding ad hoc entrustment of junior residents, including patient safety, desire to optimise the learning environment, junior resident qualities (such as discernment and communication skills), learner handover from colleagues, and situational factors. Uniquely, senior residents discussed how their role as a middle manager and their desire to protect the junior resident (from burnout, becoming a second victim and from attendings) impacts their decisions.

CONCLUSIONS

Although senior residents make ad hoc entrustment decisions with some similar considerations to attendings, they also seem to think about additional factors. It may be that these different considerations need to be accommodated in documentation of ad hoc entrustment decisions if these documents are to be used for high-stakes summative entrustment decisions made by competency committees.

Entrustment versus performance scale in high-stakes OSCEs: Rater insights and psychometric properties

BACKGROUND

Although entrustment scales are increasingly applied in workplace-based assessments, their role in OSCEs remains unclear. We investigated raters' perceptions using an entrustment scale and psychometric analyses.

METHOD

A mixed-methods design was used. OSCE raters' (n = 162) perceptions were explored via questionnaire and four focus groups (n = 14). Psychometric OSCE properties were analyzed statistically.

RESULTS

Raters (n = 53, response rate = 41%) considered the entrustment scale comprehensible (89%) and applicable (60%). A total of 43% preferred the entrustment scale, 21% preferred the global performance scale, and 36% were undecided. Raters' written comments indicated that while they appreciated the authenticity of entrustment levels, they considered them subjective. The focus groups highlighted three main themes: (1) recollections of the clinical workplace as a cognitive reference triggered by entrustment scales; (2) factors influencing entrustment decisions; and (3) cognitive load is reduced at the perceived cost of objectivity. Psychometric analyses (n = 480 students) revealed improvements in some OSCE metrics when entrustment and global performance scales were combined.

CONCLUSION

Entrustment scales are beneficial for high-stakes OSCEs and have greater clinical relevance from the raters' perspective. Our findings support the use of entrustment and global performance scales in combination.

The influence of clinical coaching teams on quality of entrustable professional activity assessments

Background

Coaching is an important component of workplace-based assessment in competency-based medical education. Longitudinal coaching relationships have been proposed to enhance the trainee-supervisor relationship and promote high-quality assessment.

Objective

The objective of this study was to determine the influence of longitudinal coaching relationships on the quality of entrustable professional activity (EPA) assessments.

Methods

EPAs (n = 174) completed by emergency medicine (EM) supervisors between July 2020 and June 2021 were extracted and divided into two groups; one group consisted of EPAs completed by supervisors when a longitudinal coaching relationship existed (n = 87) and the other group consisted of EPAs completed by the same supervisors when no coaching relationship existed (n = 87). Three physicians were recruited to rate the EPAs using the Quality of Assessment and Learning (QuAL) score, a previously published measure of EPA quality. An analysis of variance was performed to compare mean QuAL scores between the groups. Linear regression analysis was conducted to examine the relationship between trainee performance (EPA rating) and EPA assessment quality (QuAL score).

Results

All raters completed the survey. The mean ± SD QuAL score in the coaching relationship group (3.63 ± 0.91) was higher than the no coaching relationship group (3.51 ± 1.10) but the difference was not statistically significant (p = 0.40). Supervisor was a significant predictor of QuAL score (p = 0.012) and supervisor alone accounted for 26% of the variability in QuAL scores (R2 = 0.26). There was no significant relationship between trainee performance and EPA assessment quality.

Conclusions

The presence of a longitudinal coaching relationship did not influence the quality of EPA assessments.

External validation of the ultrasound competency assessment tool

Objective

Point-of-care ultrasound (POCUS) is a core component of emergency medicine (EM) residency training. No standardized competency-based tool has gained widespread acceptance. The ultrasound competency assessment tool (UCAT) was recently derived and validated. We sought to externally validate the UCAT in a 3-year EM residency program.

Methods

This was a convenience sample of PGY-1 to -3 residents. Utilizing the UCAT and an entrustment scale, as described in the original study, six different evaluators split into two groups graded residents in a simulated scenario involving a patient with blunt trauma and hypotension. Residents were asked to perform and interpret a focused assessment with sonography in trauma (FAST) examination and apply the findings to the simulated scenario. Demographics, prior POCUS experience, and self-assessed competency were collected. Each resident was evaluated simultaneously by three different evaluators with advanced ultrasound training utilizing the UCAT and entrustment scales. Intraclass correlation coefficient (ICC) between evaluators was calculated for each assessment domain; analysis of variance was used to compare UCAT performance and PGY level and prior POCUS experience.

Results

Thirty-two residents (14 PGY-1, nine PGY-2, and nine PGY-3) completed the study. Overall, ICC was 0.9 for preparation, 0.57 for image acquisition, 0.3 for image optimization, and 0.46 for clinical integration. There was moderate correlation between number of FAST examinations performed and entrustment and UCAT composite scores. There was poor correlation between self-reported confidence and entrustment and UCAT composite scores.

Conclusions

We had mixed results in our attempt to externally validate the UCAT with poor correlation between faculty and moderate to good correlation with faculty to diagnostic sonographer. More work is needed to validate the UCAT before adoption.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

Trainees' perceptions of being allowed to fail in clinical training: A sense-making model

INTRODUCTION

Clinical supervisors allow trainees to fail during clinical situations when trainee learning outweighs concerns for patient safety. Trainees perceive failure as both educationally valuable and emotionally draining; however, the nuance of supervised failures has not been researched from the trainee perspective. This study explored trainees' awareness and their experience of failure and allowed failure to understand those events in-depth.

METHODS

We interviewed 15 postgraduate trainees from nine teaching sites in Europe and Canada. Participants were a purposive sample, representing 1-10 years of clinical training in various specialties. Consistent with constructivist grounded theory, data collection and analysis were iterative, supporting theoretical sampling to explore themes.

RESULTS

Trainees reported that failure was a common, valuable, and emotional experience. They perceived that supervisors allowed failure, but they reported never having it explicitly confirmed or discussed. Therefore, trainees tried to make sense of these events on their own. If they interpreted a failure as allowed by the supervisor, trainees sought to ascertain supervisory intentions. They described situations where they judged supervisor's intentions to be constructive or destructive.

DISCUSSION

Our results confirm that trainees perceive their failures as valuable learning opportunities. In the absence of explicit conversations with supervisors, trainees tried to make sense of failures themselves. When trainees judge that they have been allowed to fail, their interpretation of the event is coloured by their attribution of supervisor intentions. Perceived intentions might impact the educational benefit of the experience. In order to support trainees' sense-making, we suggest that supervisory conversations during and after failure events should use more explicit language to discuss failures and explain supervisory intentions.

Which Entrustment-Supervision Scale is Right for Pharmacy Education?

Entrustable professional activities (EPAs) are an outgrowth of the competency-based educational model to support workplace-based learner assessments and evaluation. A learner's performance of EPAs is assessed by the degree of provided entrustment and required supervision rather than by a score, percentage, or letter grade typically assigned in traditional academic coursework. Entrustment-supervision (ES) scales are used to document learner progression and steer learner development over time. The purpose of this article is to critique various ES tools in health professions education for utilization within an EPA framework for learner assessment in workplace-based settings and to determine which will best suit pharmacy education. Exploring the advantages and disadvantages across all types of ES scales is a critical step in determining the most useful ES tool for use within a specific pharmacy institution and across the Academy. An ES scale with the traditional 5 levels, a prospective assessment frame, and increased stratification at lower levels should be recommended by the Academy and utilized in workplace-based settings for formative and summative assessment to provide more valid assessment of learners, support the ideal of life-long learning, and give more meaning for pharmacy faculty and learners within assessment.

Validity and Educational Impact of Clinical Entrustable Professional Activities Linked to the Nutrition Care Process for Work-Based Assessment of Entry-Level Dietetics Students: Evaluation of a 3-Year Implementation Study in Australia

BACKGROUND

Entrustable professional activities (EPAs) describe key workplace activities and are increasingly being used for student work-based assessment. An EPA-based work-based assessment tool offers potential to increase understanding of dietetics student skill development and opportunity for standardised work-based student assessment.

OBJECTIVE

To determine construct validity and educational impact of an EPA work-based assessment tool for dietetics placement students in clinical settings.

DESIGN

Using a time series design, supervisor and student self-assessment data collected from an EPA-based assessment tool from three cohorts of dietetics students and supervisors from 2017-2019 was analyzed.

PARTICIPANTS AND SETTING

Dietetics students (n = 145) from an accredited dietetic training program in Australia and affiliated metropolitan and rural hospitals.

MAIN OUTCOME MEASURES

Construct validity was determined through analysis of supervisor evaluation of student performance against EPAs over time. Educational impact was determined through comparing student performance across EPAs to predicted milestones to identify areas of least entrustment.

STATISTICAL ANALYSES PERFORMED

Student performance over time and differences between student self-assessment and supervisor assessment for each EPA were investigated using Wilcoxon signed-rank tests and linear mixed-model analysis. Descriptive statistics summarised student performance against each EPA.

RESULTS

Performance significantly increased over time in 35 out of 37 EPAs. Significant differences between supervisor and student self-assessment were evident in 9 out of 37 EPAs. Dietetics student performance varied across EPAs with 88.2% of students achieving entrustment for nutrition management EPAs compared with 100% for professionalism EPAs.

CONCLUSIONS

The tool's construct validity was established. EPAs identified areas of Nutrition Care Process skills development requiring additional support. Data collected by the tool can inform teaching.

A scoping review of assessment methods of competence of general surgical trainees

BACKGROUND

Only rigorous evaluation of competence will result in the production of safe surgeons that are able to provide the best care for patients. The development of competency-based assessment should ultimately be evidence driven.

OBJECTIVES

Explore the volume of existing evidence pertaining to the different objective assessment methods reported in the literature.

ELIGIBILITY CRITERIA

Studies describing objective assessment of postgraduate general surgical trainees within the last 20 years.

SOURCES OF EVIDENCE

PubMed, Ovid Medline and Web of Sciences.

CHARTING METHODS

A data chart proforma was designed and data were extracted into tables. Basic numerical analysis of extracted data and narrative synthesis of charted data.

RESULTS

A total of 343 papers were reviewed. 26 were eligible for inclusion. 92% of articles were published from 2008 onwards. 50% have been published in the last five years. The articles originated from 6 different countries, predominantly the United Kingdom (42%), followed by the United States of America (38%). In addition, a small number were published from Canada (8%), Japan (4%), Germany (4%) and Australia (4%). UK publications were predominantly between 2008 and 2014 while the USA had a later predominance between 2015 and 2018. 42% were based on quantitative methodology, 27% had a qualitative approach while 31% had mixed analysis. There were sixteen assessment methods presented. The most common type of assessment was Objective Structured Assessments (27%), which included Objective Structured Assessment of Technical Skill (OSATS) (23%) and Objective Structured Assessment of Non-Technical Skill (4%). Procedure Based Assessment (PBA) (23%) and Entrustability Scales (23%) were also prevalent.

CONCLUSIONS

This scoping review has identified a range of different assessment methods. The assessment methods with a higher volume and level of supporting evidence were OSATS, PBAs and Entrustability Scales. There was a lower volume and level of supporting evidence found within this review for the remaining assessment methods.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

PURPOSE

Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.

METHOD

This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.

RESULTS

Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.

CONCLUSIONS

Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

A Tool to Assess Competence in Critical Care Ultrasound Based on Entrustable Professional Activities

Background

Existing assessment tools for competence in critical care ultrasound (CCUS) have limited scope and interrupt clinical workflow. The framework of entrustable professional activities (EPAs) is well suited to developing an assessment tool that is comprehensive and readily integrated into the intensive care unit (ICU) training environment.

Objective

This study sought to design an EPA-based tool to assess competence in CCUS for pulmonary and critical care fellows and to assess the validity and reliability of the tool.

Methods

Eight experts in CCUS met to define the core EPAs for CCUS. A nominal group technique was used to reach consensus. An assessment tool was created based on the EPAs with a modified Ottawa entrustability scale. Trained faculty evaluated pulmonary and critical care fellows using this tool in the ICU over a 6-month study period at a single institution. An assessment of validity of the EPA-based tool is made with four sources of validity evidence: content, response process, reliability, and relation to other variables. Reliability and response process data were generated using generalizability theory analysis to estimate sources of variance in entrustment scores. Analysis of response process validity and validity by relation to other variables was performed using regression models.

Results

Fifty-four assessments were recorded during the study period, conducted on 23 trainees by 13 faculty. Content validity of the tool was demonstrated using expert consensus and published guidelines from critical care societies to define the EPAs. Response process validity was demonstrated by the low variance in entrustment scores due to evaluators (0.086 or 6%) and high agreement between score and trainee self-assessment (regression coefficient, 0.82; P < 0.0001). Reliability was demonstrated by the high "true" variance in entrustment score attributable to the trainee: 0.674 or 45%. Validity by relation to other variables was demonstrated using regression analysis to show correlation between entrustment score and the number of times a fellow has performed an EPA (regression coefficient, 0.023; P < 0.0001).

Conclusion

An EPA-based assessment tool for competence in CCUS was created. We obtained sufficient validity evidence on three of the diagnostic EPAs. Procedural EPAs were infrequently assessed, limiting generalizability in this subgroup.

Developing a CPD model for Eswatini-a participatory action research study

BACKGROUND

Continuing professional development (CPD) is a key aspect to fulfil a commitment to lifelong learning for professionals registered with the Medical and Dental Council, the intention being to promote the health of patients and develop clinical expertise. The absence of formal CPD requirements for practitioners in Eswatini has resulted in a move to introduce an accredited system.

METHODS

The qualitative study followed a participatory action research (PAR) methodology using a cooperative inquiry group of 10 medical practitioners in Eswatini to investigate how the current CPD program could be improved and formalised. PAR entailed four stages; observation, reflection, planning and action, using a semi-structured format to explore the areas of concern.

RESULTS

Reflecting on the current situation resulted in three ways to improve CPD being identified: (1) adopt a formal, compulsory CPD model; (2) recognise achievements by practitioners who endeavour to improve their skills/knowledge through Entrustable Professional Activities, and (3) ensure that CPD is relevant to the workplace by using Quality-Improvement CPD (QI-CPD) and reflective diaries. These would be done by involving local practitioners, using adult learning principles and ensuring continuous evaluation and improvement of the CPD model.

CONCLUSIONS

There was general agreement on the need for a formalised CPD system to improve skill levels and provide an open platform to enhance patient care in a resource constrained setting. The findings provided information that can be used to plan and action its implementation through engagement with the country's doctors in various forums and through ongoing research.

Advanced neonatal procedural skills: a simulation-based workshop: impact and skill decay

BACKGROUND

Trainees aiming to specialize in Neonatal Perinatal Medicine (NPM), must be competent in a wide range of procedural skills as per the Royal College of Canada. While common neonatal procedures are frequent in daily clinical practice with opportunity to acquire competence, there are substantial gaps in the acquisition of advanced neonatal procedural skills. With the advent of competency by design into NPM training, simulation offers a unique opportunity to acquire, practice and teach potentially life-saving procedural skills. Little is known on the effect of simulation training on different areas of competence, and on skill decay.

METHODS

We designed a unique simulation-based 4-h workshop covering 6 advanced procedures chosen because of their rarity yet life-saving effect: chest tube insertion, defibrillation, exchange transfusion, intra-osseus (IO) access, ultrasound-guided paracentesis and pericardiocentesis. Direct observation of procedural skills (DOPS), self-perceived competence, comfort level and cognitive knowledge were measured before (1), directly after (2), for the same participants after 9-12 months (skill decay, 3), and directly after a second workshop (4) in a group of NPM and senior general pediatric volunteers.

RESULTS

The DOPS for all six procedures combined for 23 participants increased from 3.83 to 4.59. Steepest DOPS increase pre versus post first workshop were seen for Defibrillation and chest tube insertion. Skill decay was evident for all procedures with largest decrease for Exchange Transfusion, followed by Pericardiocentesis, Defibrillation and Chest Tube. Self-perceived competence, comfort and cognitive knowledge increased for all six procedures over the four time points. Exchange Transfusion stood out without DOPS increase, largest skill decay and minimal impact on self-assessed competence and comfort. All skills were judged as better by the preceptor, compared to self-assessments.

CONCLUSIONS

The simulation-based intervention advanced procedural skills day increased preceptor-assessed directly observed procedural skills for all skills examined, except exchange transfusion. Skill decay affected these skills after 9-12 months. Chest tube insertions and Defibrillations may benefit from reminder sessions, Pericardiocentesis may suffice by teaching once. Trainees' observed skills were better than their own assessment. The effect of a booster session was less than the first intervention, but the final scores were higher than pre-intervention.

TRIAL REGISTRATION

Not applicable, not a health care intervention.

Advancements in Surgical Performance Evaluation and Feedback

PURPOSE OF REVIEW

How today's urology trainees acquire surgical skills has changed dramatically due to multiple forces placing strain on the graduate medical education mission. The development of workplace-based assessments that deliver feedback while capturing performance data has led to a paradigm shift toward individualized learning.

RECENT FINDINGS

Delivering feedback that drives surgical skill development requires the educator to provide a meaningful assessment of the learner after an operative experience. Workplace-based assessment involves direct observation of routine clinical practice and has become a central component of competency-based medical education. Urology has the chance to fully embrace competency-based medical education, employing robust feedback mechanisms and workplace-based assessments. We must first define what it means to be a proficient urologist and design an assessment system that captures this collective sentiment. This can only be done through effective engagement and collaboration with stakeholders across our specialty.

Supervising the senior medical resident: Entrusting the role, supporting the tasks

INTRODUCTION

Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how internal medicine supervisors conceptualise entrusting senior medical residents while supervising them on the acute care wards.

METHODS

Guided by constructivist grounded theory, we interviewed 19 physicians who regularly supervised senior internal medicine residents on inpatient wards at three Canadian universities. We developed a theoretical model through iterative cycles of data collection and analysis using a constant comparative process.

RESULTS

On the internal medicine ward, the senior resident role is viewed as a fundamentally managerial and rudimentary version of the supervisor's role. Supervisors come to trust their residents in the senior role through an early 'hands-on' period of assessment followed by a gradual withdrawal of support to promote independence. When considering entrustment, supervisors focused on entrusting a particular scope of the senior resident role as opposed to entrustment of individual tasks. Irrespective of the scope of the role that was entrusted, supervisors at times stepped in and stepped back to support specific tasks.

CONCLUSION

Supervisors' stepping in and stepping back to support individual tasks on the acute care ward has an inconsistent relationship to their entrustment of the resident with a particular scope of the senior resident role. In this context, entrustment-based assessment would need to capture more of the holistic perspective of the supervisor's entrustment of the senior resident role. Understanding the dance of supervision, from relatively static overall support of the resident in their role, to fluidly stepping in and out for specific patient care tasks, allows us insight into the affordances of the supervisory relationship and how it may be leveraged for assessment.

Do entrustment scales make a difference in the inter-rater reliability of the workplace-based assessment?

BACKGROUND

A workplace-based assessment (WBA) is used to assess learners' competencies in their workplaces. Many workplace assessment tools are available and validated to assess various constructs. The implementation of workplace-based assessment requires proper training of the staff.

OBJECTIVE

This study aimed to explore the impact of staff training on WBA practices and evaluate the inter-rater reliability of these practices while using entrustment scales, performance descriptors, and personal judgment.

DESIGN

A quasi-experimental study, in which the staff members of the orthopedic department were invited to participate in a training program on the use of entrustment scales and assessment descriptors within the WBA tools. As a response to the training, subjective judgment was replaced by entrustment scales and performance descriptors in a trauma course offered by the orthopedic department. The inter-rater reliability of the WBA was evaluated using various rating scales.

RESULTS

The entrustment scales had higher inter-rater reliability of the assessment tools than performance descriptors and the personal judgment.

CONCLUSION

The inter-rater reliability was highest when using entrustment scales for WBAs, which could indicate that the entrustment scales achieve good psychometric properties as regards consistency among different raters. Thus, they decrease the confounding effect of differences in assessors. They may also give a clearer image of the actual academic level of the learners.

Assessment of laparoscopic skills: comparing the reliability of global rating and entrustability tools

BACKGROUND

Competence by design (CBD) residency programs increasingly depend on tools that provide reliable assessments, require minimal rater training, and measure progression through the CBD milestones. To assess intraoperative skills, global rating scales and entrustability ratings are commonly used but may require extensive training. The Competency Continuum (CC) is a CBD framework that may be used as an assessment tool to assess laparoscopic skills. The study aimed to compare the CC to two other assessment tools: the Global Operative Assessment of Laparoscopic Skills (GOALS) and the Zwisch scale.

METHODS

Four expert surgeons rated thirty laparoscopic cholecystectomy videos. Two raters used the GOALS scale while the remaining two raters used both the Zwisch scale and CC. Each rater received scale-specific training. Descriptive statistics, inter-rater reliabilities (IRR), and Pearson's correlations were calculated for each scale.

RESULTS

Significant positive correlations between GOALS and Zwisch (r = 0.75, p < 0.001), CC and GOALS (r = 0.79, p < 0.001), and CC and Zwisch (r = 0.90, p < 0.001) were found. The CC had an inter-rater reliability of 0.74 whereas the GOALS and Zwisch scales had inter-rater reliabilities of 0.44 and 0.43, respectively. Compared to GOALS and Zwisch scales, the CC had the highest inter-rater reliability and required minimal rater training to achieve reliable scores.

CONCLUSION

The CC may be a reliable tool to assess intraoperative laparoscopic skills and provide trainees with formative feedback relevant to the CBD milestones. Further research should collect further validity evidence for the use of the CC as an independent assessment tool.

Challenges and opportunities for educating health professionals after the COVID-19 pandemic

The education of health professionals substantially changed before, during, and after the COVID-19 pandemic. A 2010 Lancet Commission examined the 100-year history of health-professional education, beginning with the 1910 Flexner report. Since the publication of the Lancet Commission, several transformative developments have happened, including in competency-based education, interprofessional education, and the large-scale application of information technology to education. Although the COVID-19 pandemic did not initiate these developments, it increased their implementation, and they are likely to have a long-term effect on health-professional education. They converge with other societal changes, such as globalisation of health care and increasing concerns of health disparities across the world, that were exacerbated by the pandemic. In this Health Policy, we list institutional and instructional reforms to assess what has happened to health-professional education since the publication of the Lancet Commission and how the COVID-19 pandemic altered the education process.

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project

BACKGROUND

Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation.

METHODS

Using a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations.

RESULTS

Between July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard.

CONCLUSIONS

We identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.

The logic behind entrustable professional activity frameworks: A scoping review of the literature

INTRODUCTION

Entrustable professional activities (EPAs), discrete profession-specific tasks requiring integration of multiple competencies, are increasingly used to help define and inform curricula of specialty training programmes. Although guidelines exist to help guide the developmental process, deciding what logic to use to draft a preliminary EPA framework poses a crucial but often difficult first step. The logic of an EPA framework can be defined as the perspective used by its developers to break down the practice of a profession into units of professional work. This study aimed to map dominant logics and their rationales across postgraduate medical education and fellowship programmes.

METHODS

A scoping review using systematic searches within five electronic databases (Medline, Embase, Google Scholar, Scopus and Web of Science) was performed. Dominant logics of included papers were identified using inductive coding and iterative analysis.

RESULTS

In total, 42 studies were included. Most studies were conducted in the United States (n = 22; 52%), Canada (n = 6; 14%) and the Netherlands (n = 4; 10%). Across the reported range of specialties, family medicine (n = 4; 10%), internal medicine (n = 4; 10%), paediatrics (n = 3; 7%) and psychiatry (n = 3; 7%) were the most common. Three dominant logics could be identified, namely, 'service provision', 'procedures' and/or 'disease or patient categories'. The majority of papers (n = 37; 88%) used two or more logics when developing EPA frameworks (median = 3, range = 1-4). Disease or patient groups and service provision were the most common logics used (39% and 37%, respectively).

CONCLUSIONS

Most programmes used a combination of logics when trying to capture the essential tasks of a profession in EPAs. For each of the three dominant logics, the authors arrived at a definition and identified benefits, limitations and examples. These findings may potentially inform best practice guidelines for EPA development.

Implementation of Entrustable Professional Activities assessments in a Canadian obstetrics and gynecology residency program: a mixed methods study

BACKGROUND

Since the implementation of competency-based medical education (CBME) across residency training programs in Canada, there has been limited research understanding how entrustable professional activity (EPA) assessments are used by faculty supervisors and residents.

OBJECTIVE

This study examines how EPA assessments are used in an Obstetrics and Gynecology residency program and the impact of implementation on both groups.

METHODS

A mixed methods study design was used. Part one involved the aggregation of descriptive data of EPA assessment completion for postgraduate year 1 and 2 residents from July 2019 to May 2020. Part two involved a thematic analysis of semi-structured interviews of residents and faculty.

RESULTS

There was significant uptake of EPA assessments across community and teaching hospitals with widespread contribution of assessment data from faculty. However, both residents and faculty reported that the intended design of EPA assessments as low-stakes assessments to provide formative feedback is not how EPA assessments are experienced. Residents and faculty noted the increased level of administrative burden and related perceived stress amongst the resident group.

CONCLUSIONS

The implementation of EPA assessments is feasible across a variety of sites. However, previous measurement challenges remain. Neither residents nor faculty perceive the value of EPAs to improve feedback, despite their intended nature.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

Does direct observation influence the quality of workplace-based assessment documentation?

Background

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect).

Methods

The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis.

Results

A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84.

Conclusions

Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

The shift from disbelieving underperformance to recognising failure: A tipping point model

CONTEXT

Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed.

METHODS

Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process.

RESULTS

The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm.

CONCLUSION

Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters

PURPOSE

In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales.

METHOD

The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments.

RESULTS

Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs.

CONCLUSIONS

In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

A Literature Review: Entrustable Professional Activities, an assessment tool for postgraduate dental training?

Assessing when dental trainees are ready to independently undertake clinical procedures at specialist level is critical for dental postgraduate programmes to determine when a trainee is 'work ready', in addition to ensuring patient safety. Entrustable professional activities (EPA) are a novel method of competency-based assessment. An EPA is a unit of professional practice or critical clinical activity identified within dental training programmes, which should be assessed during training, to establish if trainees are ready for independent practice, with a progressive decrease in supervision, based on supervisors' entrustment decisions. This article describes EPAs, entrustment decisions, including entrustment supervision scales and the process recommended to develop EPAs within dental curricula. EPAs have not been formally introduced for assessment within dental education programmes in the United Kingdom, but recent developments have been described in undergraduate dental education globally. Clinical significance: Competency-based assessments need to be continually developed to adapt to rapidly changing population health care and dental needs, to determine when dental trainees are ready for independent clinical practice. Early development of entrustable professional activities for assessment in undergraduate dental programmes has been well received by both trainees and supervisors. Further investigation is required to consider formal development of EPAs within postgraduate dental programmes.

Veterinary undergraduates' attitudes to and understanding of clinical skills learning

BACKGROUND

The study investigated student understanding of development of their clinical skills learning, focusing on personal experiences of the learning strategies they employ and their experiences of judging their own competency, integrating skills and transferring skill knowledge.

METHODS

Twenty-nine veterinary students from two veterinary schools took part in semi-structured interviews investigating their perceptions of clinical skills importance and development. Thematic analysis of transcribed interviews was used for data analysis.

RESULTS

Themes that were developed were: (1) Students understand clinical skills to be individual technical practical skills but also understand how skills are combined to achieve complex tasks. (2) Students understand the concept of competency development and learn skills, review their progress and self-audit their skill competency in similar ways throughout the course. (3) Personal confidence, success and positive communication with peers and supervisors are critical to learn clinical skills.

CONCLUSION

Students understand clinical skills learning and use reflective skills widely to track developing skills competency. Prioritising individual feedback within practical sessions and training on integration of skills and communicating with supervisors ahead of clinical placements would be beneficial. Training in clinical skills study techniques, reflection and resilience will enable students to be proactive and to manage their feelings around uncertainty.

Inter-Rater and Intra-Rater Reliability of the INSPECT (Interactive Nutrition Specific Physical Exam Competency Tool) Measured in Multi-Site Acute Care Settings

Even as new medical modalities, diagnostics, and technologies are rapidly changing healthcare, providing patients with safe, high-quality care remains the central focus. To provide safe patient care, healthcare providers are obligated to demonstrate and maintain the necessary competence. As more healthcare disciplines move toward a competency-based education model, it is essential to extend the competence verification from the academic educational level to the patient’s bedside. The nutrition-focused physical exam (NFPE) is a competency recently adopted by registered dietitian nutritionists (RDNs) for assessing patients’ nutritional status. Being a newly acquired skill, validated tools are required to measure NFPE competence during routine clinical practice. The Interactive Nutrition Specific Physical Exam Competency Tool (INSPECT) is a new tool developed specifically to observe and measure RDNs’ NFPE competence in clinical settings. The INSPECT was designed and validated for content using expert RDNs’ input in the first and second phases of the study. This current study aimed to assess the reliability of the INSPECT through multi-site observations by clinical supervisors evaluating RDNs’ NFPE competency during patient assessment. The INSPECT exhibited good inter-rater reliability (ICC = 0.78 for the first assessment and ICC = 0.68 for the second assessment), moderate to strong intra-rater reliability for 37 of 41 items (Spearman rho = 0.54 to 1.0), and excellent internal consistency (Cronbach’s α = 0.86 for the first assessment and α = 0.92 for the second assessment). In total, 10 out of the 11 INSPECT subsets showed good to excellent internal consistency (α ranging from 0.70 to 0.98). The results demonstrate that the INSPECT is a reliable tool, is stable over time, and has good agreement and excellent consistency between raters. The INSPECT can be a valuable tool to measure, promote and maintain RDNs’ NFPE competence in authentic acute care settings.

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

BACKGROUND

Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training.

METHODS

The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences.

RESULTS

We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test-retest reliability was high (intraclass correlation coefficient [ICC_2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent.

CONCLUSIONS

This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.