Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study

BACKGROUND

Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs.

METHODS

The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability.

RESULTS

Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability.

CONCLUSION

The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

COVID-19 related disruptions to medical education and perceived clinical capability of new resident physicians: a nationwide study of over 1200 first-year residents

The COVID-19 pandemic transformed the final year of undergraduate medical education for thousands of medical students across the globe. Out of concern for spreading SARS-CoV-2 and conserving personal protective equipment, many students experienced declines in bedside clinical exposures. The perceived competency of this class within the context of the pandemic is unclear. We designed and distributed a survey to measure the degree to which recent medical school graduates from the USA felt clinically prepared on 13 core clinical skills. Of the 1283 graduates who matched at HCA Healthcare facilities, 90% (1156) completed the survey. In this national survey, most participants felt they were competent in their clinical skills. However, approximately one out of four soon-to-be residents felt they were clinically below where they should be with regard to calling consultations, performing procedures, and performing pelvic and rectal exams. One in five felt they were below where they should be with regard to safely transitioning care. These perceived deficits in important skill sets suggest the need for evaluation and revised educational approaches in these areas, especially when traditional in-person practical skills teaching and practice are disrupted.

Would I Trust or Will I Trust? The Gap between Entrustment Determinations and Entrustment Decisions for Trainees in Pharmacy and Other Health Professions

Entrustable Professional Activities (EPAs) and entrustment decision making are rapidly becoming mainstream in competency-based education in the health professions. EPAs are the units of professional practice to entrust graduates with once they have developed the required competencies. They were conceived to enable a gradual increase in professional autonomy during training, by allowing trainees to practice activities in which they have demonstrated they have mastered well, with decreasing supervision. However, practicing health care unsupervised generally requires licensure. The question for pharmacy education, as well as for undergraduate medical education, is can students be given any autonomy in practice, even when they have fully mastered an EPA yet remain unlicensed? While entrustment decisions for licensed practitioners have autonomy consequences, some educators in undergraduate programs speak of 'entrustment determinations', to avoid decisions about students that affect patient care, in other words saying, we would trust you, rather than we will trust you. However, graduating learners without the experience of responsibility and reasonable autonomy creates a gap with full practice responsibilities, which may jeopardize patient safety after training. What can programs do to retain the power of using EPAs while at the same time guarding patient safety?

Specialty-Specific Entrustable Professional Activities: A Bridge to Internship

Background Undergraduate medical education aims to prepare learners to become capable residents. New interns are expected to perform clinical tasks with distant supervision reliant on having acquired a medical degree. However, there is limited data to discuss what entrustment residency programs grant versus what the medical schools believe they have trained their graduates to perform. At our institution, we sought to foster an alliance between undergraduate medical education (UME) and graduate medical education (GME) toward specialty-specific entrustable professional activities (SSEPAs). These SSEPAs create a bridge to residency and help students structure the final year of medical school while striving for entrustability for day one of residency. This paper describes the SSEPA curriculum development process and student self-assessment of competence. Methodology We piloted an SSEPA program with the departments of Family Medicine, Internal Medicine, Neurology, and Obstetrics & Gynecology. Utilizing Kern's curriculum development framework, each specialty designed a longitudinal curriculum with a post-match capstone course. Students participated in pre-course and post-course self-assessments utilizing the Chen scale for each entrustable professional activity (EPA). Results A total of 42 students successfully completed the SSEPA curriculum in these four specialties. Students' self-assessed competence levels rose from 2.61 to 3.65 in Internal Medicine; 3.23 to 4.12 in Obstetrics and Gynecology; 3.62 to 4.13 in Neurology; and 3.65 to 3.79 in Family Medicine. Students across all specialties noted an increase in confidence from 3.45 to 4.38 in Internal Medicine; 3.3 to 4.6 in Obstetrics and Gynecology; 3.25 to 4.25 in Neurology; and 4.33 to 4.67 in Family Medicine. Conclusions A specialty-specific curriculum utilizing a competency-based framework for learners traversing the UME to GME journey in the final year of medical school improves learner confidence in their clinical abilities and may lead to an improved educational handoff between UME and GME.