The ingredients of a rich entrustment decision

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

When words are your scalpel, what and how information is exchanged may be differently salient to assessors

PURPOSE

Variable assessments of learner performances can occur when different assessors determine different elements to be differently important or salient. How assessors determine the importance of performance elements has historically been thought to occur idiosyncratically and thus be amenable to assessor training interventions. More recently, a main source of variation found among assessors was two underlying factors that were differently emphasised: medical expertise and interpersonal skills. This gave legitimacy to the theory that different interpretations of the same performance may represent multiple truths. A faculty development activity introducing assessors to entrustable professional activities in which they estimated a learner's level of readiness for entrustment provided an opportunity to qualitatively explore assessor variation in the context of an interaction and in a setting in which interpersonal skills are highly valued.

METHODS

Using a constructivist grounded theory approach, we explored variation in assessment processes among a group of palliative medicine assessors who completed a simulated direct observation and assessment of the same learner interaction.

RESULTS

Despite identifying similar learner strengths and areas for improvement, the estimated level of readiness for entrustment varied substantially among assessors. Those who estimated the learner as not yet ready for entrustment seemed to prioritise what information was exchanged and viewed missed information as performance gaps. Those who estimated the learner as ready for entrustment seemed to prioritise how information was exchanged and viewed the same missed information as personal style differences or appropriate clinical judgement. When presented with a summary, assessors expressed surprise and concern about the variation.

CONCLUSION

A main source of variation among our assessors was the differential salience of performance elements that align with medical expertise and interpersonal skills. These data support the theory that when assessing an interaction, differential salience for these two factors may be an important and perhaps inevitable source of assessor variation.

Feedback in an Entrustment-Based Objective Structured Clinical Examination: Analysis of Content and Scoring Methods

Background The integration of entrustable professional activities (EPAs) within objective structured clinical examinations (OSCEs) has yielded a valuable avenue for delivering timely feedback to residents. However, concerns about feedback quality persist. Objective This study aimed to assess the quality and content alignment of verbal feedback provided by examiners during an entrustment-based OSCE. Methods We conducted a progress test OSCE for internal medicine residents in 2022, assessing 7 EPAs. The immediate 2-minute feedback provided by examiners was recorded and analyzed using the Quality of Assessment of Learning (QuAL) score. We also analyzed the degree of alignment with EPA learning objectives: competency milestones and task-specific abilities. In a randomized crossover experiment, we compared the impact of 2 scoring methods used to assess residents' clinical performance (3-point entrustability scales vs task-specific checklists) on feedback quality and alignment. Results Twenty-one examiners provided feedback to 67 residents. The feedback demonstrated high quality (mean QuAL score 4.3 of 5) and significant alignment with the learning objectives of the EPAs. On average, examiners addressed in their feedback 2.5 milestones (61%) and 1.2 task-specific abilities (46%). The scoring methods used had no significant impact on QuAL scores (95% CI -0.3, 0.1, P=.28), alignment with competency milestones (95% CI -0.4, 0.1, P=.13), or alignment with task-specific abilities (95% CI -0.3, 0.1, P=.29). Conclusions In our entrustment-based OSCE, examiners consistently offered valuable feedback aligned with intended learning outcomes. Notably, we explored high-quality feedback and alignment as separate dimensions, finding no significant impact from our 2 scoring methods on either aspect.

Assessment of Entrustable Professional Activities Among Dutch Endocrine Supervisors

Entrustable Professional Activities (EPAs) are an important tool to support individualisation of medical training in a competency-based setting and are increasingly implemented in the clinical speciality training for endocrinologist. This study aims to assess interrater agreement and factors that potentially impact EPA scores. Five known factors that affect entrustment decisions in health profesions training (capability, integrity, reliability, humility, agency) were used in this study. A case-vignette study using standardised written cases. Case vignettes (n = 6) on the topics thyroid disease, pituitary disease, adrenal disease, calcium and bone disorders, diabetes mellitus, and gonadal disorders were written by two endocrinologists and a medical education expert and assessed by endocrinologists experienced in the supervision of residents in training. Primary outcome is the inter-rater agreement of entrustment decisions for endocrine EPAs among raters. Secondary outcomes included the dichotomous interrater agreement (entrusted vs. non-entrusted), and an exploration of factors that impact decision-making. The study protocol was registered and approved by the Ethical Review Board of the Netherlands Association for Medical Education (NVMO-ERB # 2020.2.5). Nine endocrinologists from six different academic regions participated. Overall, the Fleiss Kappa measure of agreement for the EPA level was 0.11 (95% CI: 0.03-0.22) and for the entrustment decision 0.24 (95% CI 0.11-0.37). Of the five features that impacted the entrustment decision, capability was ranked as the most important by a majority of raters (56%-67%) in every case. There is a considerable discrepancy between the EPA levels assigned by different raters. These findings emphasise the need to base entrustment decisions on multiple observations, made by a team of supervisors and enriched with factors other than direct medical competence.

Feeling the responsibility: Exploring the emotional experiences of final-year medical students when carrying out clinical tasks

PURPOSE

The concept of Entrustable Professional Activities (EPA) is increasingly used to operationalize learning in the clinical workplace, yet little is known about the emotions of learners feeling the responsibility when carrying out professional tasks.

METHODS

We explored the emotional experiences of medical students in their final clerkship year when performing clinical tasks. We used an online reflective diary. Text entries were analysed using inductive-deductive content analysis with reference to the EPA framework and the control-value theory of achievement emotions.

RESULTS

Students described a wide range of emotions related to carrying out various clinical tasks. They reported positive-activating emotions, ranging from enjoyment to relaxation, and negative-deactivating emotions, ranging from anxiety to boredom. Emotions varied across individual students and were related to the characteristics of a task, an increasing level of autonomy, the students' perceived ability to perform a task and the level of supervision provided.

DISCUSSION

Emotions are widely present and impact on the workplace learning of medical students which is related to key elements of the EPA framework. Supervisors play a key role in eliciting positive-activating emotions and the motivation to learn by providing a level of supervision and guidance appropriate to the students' perceived ability to perform the task.

American Dental Education Association Compendium Entrustable Professional Activities Workgroup report

PURPOSE

Entrustable professional activities (EPAs) are discrete clinical tasks that can be evaluated to help define readiness for independent practice in the health professions and are intended to increase trust in the dental graduate. EPAs provide a framework that bridges competencies to clinical practice. This report describes the work of the American Dental Education Association (ADEA) Compendium EPA Workgroup to develop a list of EPAs for dental education and supportive resources, including specifications and a glossary.

METHODS

Preliminary work including literature and resource review, mapping of existing competencies, and review of other health professions' EPAs informed the development of our EPAs list. Workgroup members achieved consensus using a modified Delphi process. A Qualtrics survey using a validated rubric for the assessment of EPAs as described in peer-reviewed literature was used. Dental educators, including academic deans, were surveyed for feedback on the content and format of the EPAs.

RESULTS

Based on findings in the literature analysis of existing EPAs and competencies in health professions, a list of EPAs was developed along with a description of specifications. The EPA workgroup (nine members from multiple institutions) used the Delphi process in receiving feedback from various experts. A list of 11 core EPAs was vetted by dental educators including academic deans (n = ∼23), and the process of development was reviewed by EPAs experts outside dental education. A glossary was developed to align language.

CONCLUSION

These EPAs define the scope of dental practice. This report represents Phase 1 of the EPA framework development and vetting process. Future directions will include a broader vetting of the EPA list, faculty development, and national standardized technology that support this work to optimize implementation.

Big five personality traits of medical students and workplace performance in the final clerkship year using an EPA framework

BACKGROUND

The qualities of trainees play a key role in entrustment decisions by clinical supervisors for the assignments of professional tasks and levels of supervision. A recent body of qualitative research has shown that in addition to knowledge and skills, a number of personality traits are relevant in the workplace; however, the relevance of these traits has not been investigated empirically. The aim of this study was to analyse the workplace performance of final-year medical students using an Entrustable Professional Activity (EPA) framework in relation to their personality traits.

METHODS

Medical students at the end of their final clerkship year were invited to participate in an online survey-based, cross-sectional field study. In the survey, the workplace performance was captured using a framework consisting of levels of experienced supervision and a defined set of 12 end-of-undergraduate medical training EPAs. The Big Five personality traits (extraversion, agreeableness, conscientiousness, neuroticism, and openness) of the participating medical students were measured using the Big Five Inventory-SOEP (BFI-S), which consists of 15 items that are rated on a seven-point Likert scale. The data were analysed using descriptive and inferential statistics.

RESULTS

The study included 880 final-year medical students (mean age: 27.2 years, SD = 3.0; 65% female). The levels of supervision under which the final-year clerkship students carried out the EPAs varied considerably. Significant correlations were found between the levels of experienced supervision and all Big Five dimensions The correlations with the dimensions of extraversion, agreeableness, conscientiousness and openness were positive, and that for the neuroticism dimension was negative (range r = 0.17 to r = - 0.23). Multiple regression analyses showed that the combination of the Big Five personality traits accounted for 0.8-7.5% of the variance in supervision levels on individual EPAs.

CONCLUSIONS

Using the BFI-S, we found that the levels of supervision on a set of end-of-undergraduate medical training EPAs were related to the personality traits of final-year medical students. The results of this study confirm the existing body of research on the role of conscientiousness and extraversion in entrustment decision-making and, in particular, add the personality trait of neuroticism as a new and relevant trainee quality to be considered.

Level of Supervision for the Entrustable Professional Activities Common to General Pediatrics and the Subspecialties Decreases from Residency to Fellowship

OBJECTIVES

To compare level of supervision (LOS) ratings of graduating pediatric residents with their assessments as fellows for the five Entrustable Professional Activities (EPAs) common to general pediatrics and the subspecialties and to determine if the difference between ratings from residency to fellowship is less for the QI and Practice Management EPAs, since the skills needed to perform these may be less context-dependent.

METHODS

We compared ratings of graduating residents with their assessments as fellows using LOS data from two sequential EPA studies.

RESULTS

There were 65 ratings from 41 residents at the first fellow assessment. At graduation, most residents needed little to no supervision for all EPAs with 94% (61/65) of ratings level four or five. In contrast, only 5/65 (8%) of the first fellow assessments were level four or five. The ratings difference for the QI and Practice Management EPAs was similar to the others.

CONCLUSIONS

LOS ratings for the EPAs common to generalists and subspecialists reset as residents become fellows. There was no evidence that the QI and Practice Management EPAs are less context-dependent. This study provides additional validity evidence for using these LOS scales to assess trainees in pediatric residency and fellowship.

Clinical supervisors' experience of a first-time application of entrustable professional activities in clinical supervision of medical students: findings from a Swedish pilot study

BACKGROUND

Recently, all medical universities in Sweden jointly developed a framework for Entrustable Professional Activities (EPAs) for work-based training and assessment. This framework is now being introduced nationally in the new 6-year undergraduate medical programme that directly lead to a licence to practise. When EPAs are introduced, it is of central importance to gain clinical supervisors' acceptance to apply the framework in their supervision of students. The aim of this study was therefore to investigate how clinical supervisors, not familiar with EPAs, experience clinical supervision using the framework for EPAs.

METHODS

We used a purposive sampling to recruit clinical supervisors. They were given written information on EPAs with a selection of suitable EPAs and the Swedish observation rating scale for assessment of autonomy, and they were offered to attend a 30-minute introductory web course. The participants were informed that EPAs were to be tested, and the students were asked to participate. After the study period the clinical supervisors participated in semi-structured interviews. Inductive qualitative content analysis was used to analyse the transcribed interviews.

RESULTS

Three general themes emerged in the qualitative analysis: Promoting Feedback, Trusting Assessments and Engaging Stakeholders. The participants described benefits from using EPAs, but pointed out a need for preparation and adaptation to facilitate implementation. The structure was perceived to provide structured support for feedback, student involvement, entrustment decisions, enabling supervisors to allow the students to do more things independently, although some expressed caution to rely on others' assessments. Another concern was whether assessments of EPAs would be perceived as a form of examination, steeling focus from formative feedback. To understand the concept of EPA, the short web-based course and written information was regarded as sufficient. However, concern was expressed whether EPA could be applied by all clinical supervisors. Involvement and adaption of the workplace was pointed out as important since more frequent observation and feedback, with documentation requirements, increase the time required for supervision.

CONCLUSIONS

EPAs were accepted as beneficial, promoting structured feedback and assessments of the students' autonomy. Preparation of supervisors and students as well as involvement and adaptation of the workplace was pointed out as important.

Scoping review: exploring residents' views of supervisor entrustment and its effect on learning and professional development

This scoping review aimed to systematically explore the complex and nuanced perceptions of residents entrusted with significant responsibilities by their supervisors in the clinical context. We conducted a comprehensive search strategy targeting six major electronic databases. Data were extracted using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist and a scoping review protocol. Resident perceptions regarding entrustment and its impact on learning and professional development were categorized. This review elucidated the role of supervisor entrustment in shaping residents' clinical performance, notably influencing autonomy, self-confidence, responsibility for patients, learning needs and goals, uncertainty management, and professional identity formation. Residents identified factors that promote and hinder performance and entrustment development resulting from supervisor entrustment and distinguished between appropriate supervision and over- or under-supervision. Our review suggests that residents' perceptions of supervisor entrustment, as bestowed by supervisors, serves as a catalyst for enhancing autonomy, thereby exerting a significant influence on clinical performance, and fostering the development of a professional identity. However, residents' views on appropriate supervision differ in some ways from supervisors' perspectives. Further research is needed to bridge the gap between residents' expectations of autonomy and what they are given, and to explore the intricate relationship between autonomy and entrustment.

Thresholds of becoming: an exploration of threshold concepts as a means to support professional identity formation in competency-based curricula

Inherent in every clinical preceptor's role is the ability to understand the learning needs of individual trainees, enabling them to meet their potential. Competency-based medical education frameworks have been developed to this end, but efforts to identify behaviours and activities that define competence are based on mapping knowledge, skills and ability, which can be difficult to integrate into a comprehensive picture of who the trainee is becoming. Professional identity formation, in contrast, prioritizes attention to who trainees are becoming, but provision of detailed guidance to preceptors on how to best support this form of development is challenging. The tension that results limits our ability to optimally support learners as strengths in competency development may mask professional identity development gaps and vice versa. To address this tension, this paper examines how the theory of threshold concepts - troublesome ideas that, once appreciated, fundamentally change how you understand and approach a particular activity - can shine light on professional identity formation and its relationship with developing competence. The recognition and identification of threshold concepts is offered as a means to improve our ability to identify, discuss and support behaviours and actions that impact the learner's capacity to act competently as they develop their identity at various stages of training.

Assessing supervisor versus trainee viewpoints of entrustment through cognitive and affective lenses: an artificial intelligence investigation of bias in feedback

The entrustment framework redirects assessment from considering only trainees' competence to decision-making about their readiness to perform clinical tasks independently. Since trainees and supervisors both contribute to entrustment decisions, we examined the cognitive and affective factors that underly their negotiation of trust, and whether trainee demographic characteristics may bias them. Using a document analysis approach, we adapted large language models (LLMs) to examine feedback dialogs (N = 24,187, each with an associated entrustment rating) between medical student trainees and their clinical supervisors. We compared how trainees and supervisors differentially documented feedback dialogs about similar tasks by identifying qualitative themes and quantitatively assessing their correlation with entrustment ratings. Supervisors' themes predominantly reflected skills related to patient presentations, while trainees' themes were broader-including clinical performance and personal qualities. To examine affect, we trained an LLM to measure feedback sentiment. On average, trainees used more negative language (5.3% lower probability of positive sentiment, p < 0.05) compared to supervisors, while documenting higher entrustment ratings (+ 0.08 on a 1-4 scale, p < 0.05). We also found biases tied to demographic characteristics: trainees' documentation reflected more positive sentiment in the case of male trainees (+ 1.3%, p < 0.05) and of trainees underrepresented in medicine (UIM) (+ 1.3%, p < 0.05). Entrustment ratings did not appear to reflect these biases, neither when documented by trainee nor supervisor. As such, bias appeared to influence the emotive language trainees used to document entrustment more than the degree of entrustment they experienced. Mitigating these biases is nonetheless important because they may affect trainees' assimilation into their roles and formation of trusting relationships.

Medical competence as a multilayered construct

BACKGROUND

The conceptualisation of medical competence is central to its use in competency-based medical education. Calls for 'fixed standards' with 'flexible pathways', recommended in recent reports, require competence to be well defined. Making competence explicit and measurable has, however, been difficult, in part due to a tension between the need for standardisation and the acknowledgment that medical professionals must also be valued as unique individuals. To address these conflicting demands, a multilayered conceptualisation of competence is proposed, with implications for the definition of standards and approaches to assessment.

THE MODEL

Three layers are elaborated. This first is a core layer of canonical knowledge and skill, 'that, which every professional should possess', independent of the context of practice. The second layer is context-dependent knowledge, skill, and attitude, visible through practice in health care. The third layer of personalised competence includes personal skills, interests, habits and convictions, integrated with one's personality. This layer, discussed with reference to Vygotsky's concept of Perezhivanie, cognitive load theory, self-determination theory and Maslow's 'self-actualisation', may be regarded as the art of medicine. We propose that fully matured professional competence requires all three layers, but that the assessment of each layer is different.

IMPLICATIONS

The assessment of canonical knowledge and skills (Layer 1) can be approached with classical psychometric conditions, that is, similar tests, circumstances and criteria for all. Context-dependent medical competence (Layer 2) must be assessed differently, because conditions of assessment across candidates cannot be standardised. Here, multiple sources of information must be merged and intersubjective expert agreement should ground decisions about progression and level of clinical autonomy of trainees. Competence as the art of medicine (Layer 3) cannot be standardised and should not be assessed with the purpose of permission to practice. The pursuit of personal excellence in this level, however, can be recognised and rewarded.

Entrustment decision-making in genetic counseling supervision: Exploring supervisor and student perspectives to enhance training practices

Entrustment decisions are an essential part of genetic counseling supervision and have the potential to influence a student's progression toward autonomy. However, there is often uncertainty among supervisors regarding how and when to make these decisions and very few studies have examined the impact of these decisions on students. This study utilized a mixed methods approach including surveys of genetic counseling supervisors (n = 76) and students (n = 86) as well as qualitative interviews with genetic counseling supervisors (n = 20) and students (n = 20) that explored factors that influence the entrustment decisions of genetic counseling supervisors and their effect on genetic counseling students. Genetic counseling supervisors and students were recruited from various organizations across the United States and Canada and represented a range of geographic regions, hospital systems, and genetic counseling programs. A hybrid process of deductive and inductive coding and thematic analysis was used to evaluate and interpret transcripts from the supervisor and student interviews. All participants identified advantages of increased autonomy during training. However, many supervisors reported low entrustment, seldom allowing students to complete unsupervised sessions or supervised cases without interruption. Entrustment decisions were heavily influenced by student ability and confidence, as well as patient feedback. Students emphasized the negative impact of decreased entrustment on their confidence and described clear benefits to increased autonomy before, during, and after the genetic counseling appointment. Supervisors identified various barriers to entrustment pertaining to the student, clinical setting, and the patient, whereas students more often emphasized barriers pertaining to themselves. Our results highlight a tension between the clear advantages of increased entrustment and autonomy and various barriers to the provision of these opportunities. Additionally, our data suggest several ways to enhance the supervisor-student relationship and promote additional learning opportunities to support student-centered supervision.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Automated Analysis of Preceptor Comments: A Pilot Study Using Sentiment Analysis to Identify Potential Student Issues in Experiential Education

OBJECTIVE

The purpose of this paper is to describe a sentiment analysis program that aids in identifying pharmacy students at risk for progression issues by automatically scoring preceptor comments as positive or negative.

METHODS

An R-based program to analyze advanced pharmacy practice experiences and introductory pharmacy practice experiences midpoint evaluation of preceptor comments was piloted in phase 1 by comparing the sentiment analysis algorithm results to human coding. The algorithm was refined in phase 2. In phase 3, the validation phase, the final sentiment analysis algorithm analyzed all midpoint student evaluations (n = 1560). Sentiment scores were generated for each preceptor comment, and correlations were performed between sentiment scores and the quantitative scoring provided on the assessment.

RESULTS

In phase 1, agreement between faculty coders and sentiment analysis was 96%, and in phase 2, agreement between the final codes and sentiment analysis was 92.4% once keywords were added to the sentiment dictionary. In phase 3, a total of 3919 comments from 1560 evaluations were analyzed, and overall, the sentiment analysis results aligned with the quantitative data.

CONCLUSION

This sentiment analysis algorithm was accurate in capturing positive and negative comments corresponding to pharmacy student performance. Given the accuracy of this preliminary validation for flagging preceptor comments, there are numerous implications when considering the use of sentiment analysis in pharmacy education. Using a sentiment analysis program minimizes the number of qualitative preceptor comments needing review by experiential faculty, as this program can aid in identifying students at risk of progression issues.

How much is enough? Proposing achievement thresholds for core EPAs of graduating medical students in Canada

PURPOSE

The transition towards Competency-Based Medical Education at the Cumming School of Medicine was accelerated by the reduced clinical time caused by the COVID-19 pandemic. The purpose of this study was to define a standard protocol for setting Entrustable Professional Activity (EPA) achievement thresholds and examine their feasibility within the clinical clerkship.

METHODS

Achievement thresholds for each of the 12 AFMC EPAs for graduating Canadian medical students were set by using sequential rounds of revision by three consecutive groups of stakeholders and evaluation experts. Structured communication was guided by a modified Delphi technique. The feasibility/consequence models of these EPAs were then assessed by tracking their completion by the graduating class of 2021.

RESULTS

The threshold-setting process resulted in set EPA achievement levels ranging from 1 to 8 across the 12 AFMC EPAs. Estimates were stable after the first round for 9 of 12 EPAs. 96.27% of EPAs were successfully completed by clerkship students despite the shortened clinical period. Feasibility was predicted by the slowing rate of EPA accumulation overtime during the clerkship.

CONCLUSION

The process described led to consensus on EPA achievement thresholds. Successful completion of the assigned thresholds was feasible within the shortened clerkship.[Box: see text].

Residents as supervisors: How senior residents make ad hoc entrustment decisions

BACKGROUND

As residency programmes transition to competency-based medical education, there has been substantial inquiry into understanding how ad hoc entrustment decisions are made by attending supervisors in the clinical context. However, although attendings are ultimately responsible for the decisions and actions of resident trainees, senior residents are often the ones directly supervising junior residents enrolled in competency-based training programmes. This clinical dynamic has been largely overlooked in the ad hoc entrustment literature. The purpose of this study was to explore the considerations senior residents entertain when making ad hoc entrustment decisions for their junior resident colleagues.

METHODS

In semi-structured interviews, 11 senior resident supervisors (third, fourth and fifth year) in obstetrics and gynaecology described how they entrust junior residents with clinical activities in the moment. Following constructivist grounded theory methodology, data were iteratively collected and coded with constant comparison until theoretical sufficiency was determined.

RESULTS

Senior residents described many similar considerations as attendings regarding ad hoc entrustment of junior residents, including patient safety, desire to optimise the learning environment, junior resident qualities (such as discernment and communication skills), learner handover from colleagues, and situational factors. Uniquely, senior residents discussed how their role as a middle manager and their desire to protect the junior resident (from burnout, becoming a second victim and from attendings) impacts their decisions.

CONCLUSIONS

Although senior residents make ad hoc entrustment decisions with some similar considerations to attendings, they also seem to think about additional factors. It may be that these different considerations need to be accommodated in documentation of ad hoc entrustment decisions if these documents are to be used for high-stakes summative entrustment decisions made by competency committees.

Preceptor perceptions of a redesigned entrustable professional activity (EPA) assessment tool in pharmacy practice experiences

INTRODUCTION

The purpose of this study was to determine pharmacy preceptor perceptions of an entrustable professional activity (EPA) evaluation tool redesigned in 2019 and launched for use in May 2020.

METHODS

Participants received an optional survey in June 2021 regarding their perceptions of the redesigned EPA assessment tool. The survey used a combination of open- and closed-ended questions regarding preceptor perceptions of the revised form, including perceptions of accuracy, usability, ease of use, grading, EPAs, and form layout. Preceptor perceptions were measured using seven-point Likert-type items and analyzed with descriptive statistics. Open-ended items were coded by at least two faculty members using qualitative content analysis.

RESULTS

We received 106 responses (19.5% response rate) and analyzed a total of 91 responses. Most questions related to perceived accuracy were rated positively, and preceptors indicated it was easier to assign grades and provide feedback on EPAs using the new form. Preceptors agreed that EPAs indicated when students on clinical rotations excel and when they need to improve. Preceptors reported they could effectively evaluate students and do so in a way that saves them time. Overall, preceptors agreed that the form instructions helped them accurately assess student performance, and the new narratives helped them remember the assessment standards.

CONCLUSIONS

Overarchingly, preceptors perceived that the new EPA assessment tool allowed them to provide feedback and grades more accurately, efficiently, effectively, and easily. This information will assist our institution in determining next best steps for EPA assessment tool updates.

Preceptors' considerations when entrusting professional activities to postgraduate nursing students: A qualitative study

BACKGROUND

Nurse-preceptors regularly struggle to evaluate students' readiness to take care of patients unsupervised, even with sophisticated workplace-based assessment tools. Preceptors' gut feelings are not always captured well, but are critical for judgement of readiness for learner entrustment with care tasks. Studies in medical education report features that clinicians consider important when trusting students with clinical responsibilities that might also apply in nursing.

OBJECTIVES

To unravel preceptors' considerations when entrusting professional activities to postgraduate nursing students. The findings may contribute to the improvement of workplace-based assessments and the training of preceptors.

METHODS

Thematic analysis of semi-structured interviews with sixteen nurse-preceptors from three postgraduate nursing specialisations in Dutch hospitals.

RESULTS

Three themes emerged: CONCLUSIONS: For preceptors of postgraduate nursing students, entrustment requires more than merely insight into objectively measurable competencies. Entrusting is accompanied by subjectivity related to what preceptors expected of students. These expectations are in line with suggested factors in the literature-capability, integrity, reliability, agency, and humility-considered before entrusting students with clinical responsibilities identified in medical training. Entrusting is also accompanied by what preceptors realise about their own role in entrustment decisions. Combining different information sources made assessment more transparent and the implicit more explicit.

Competency-based postgraduate training: Mentoring and monitoring using entrustable professional activities with descriptive rubrics for objectivity- A step beyond Dreyfus

Background At Sri Balaji Vidyapeeth, a competency-based learning and training (CoBaLT©) model for postgraduate (PG) curriculum, within the regulatory norms, was implemented in 2016 after adequate faculty development programmes. This focused on well-defined outcomes. Methods A review of the outcomes was undertaken in 2018 as part of internal quality assurance receiving feedback from all stakeholders, viz. students, alumni and faculty. Recent publications were also reviewed. A major problem identified was lack of clarity in definition of levels leading to subjectivity in assessment while grading. It was also noted that the process needed to be aligned with the programme outcomes. Further refinements were, therefore, made to align and objectivize formative assessment using entrustable professional activities (EPA) with the aid of descriptive rubrics of sub-competencies and milestones. Addition of detailed rubrics for objectivity takes assessment a step beyond the Dreyfus model, aligning overall to the programme outcomes. Results Achievement of appropriate grades in EPAs by individual candidates ensures entrustability in professional activities by the time of completion of the PG programme. The modification was found more transparent and objective with reference to grading by the teachers and more conducive to reflections by the residents on their performance and how to improve it. Conclusions The use of descriptive rubrics along with EPAs brings transparency and plays a key role as an objective assessment tool, which can lend direction to individual resident learning and entrustability. This is an important component of outcome-based education.

Trainees' perceptions of being allowed to fail in clinical training: A sense-making model

INTRODUCTION

Clinical supervisors allow trainees to fail during clinical situations when trainee learning outweighs concerns for patient safety. Trainees perceive failure as both educationally valuable and emotionally draining; however, the nuance of supervised failures has not been researched from the trainee perspective. This study explored trainees' awareness and their experience of failure and allowed failure to understand those events in-depth.

METHODS

We interviewed 15 postgraduate trainees from nine teaching sites in Europe and Canada. Participants were a purposive sample, representing 1-10 years of clinical training in various specialties. Consistent with constructivist grounded theory, data collection and analysis were iterative, supporting theoretical sampling to explore themes.

RESULTS

Trainees reported that failure was a common, valuable, and emotional experience. They perceived that supervisors allowed failure, but they reported never having it explicitly confirmed or discussed. Therefore, trainees tried to make sense of these events on their own. If they interpreted a failure as allowed by the supervisor, trainees sought to ascertain supervisory intentions. They described situations where they judged supervisor's intentions to be constructive or destructive.

DISCUSSION

Our results confirm that trainees perceive their failures as valuable learning opportunities. In the absence of explicit conversations with supervisors, trainees tried to make sense of failures themselves. When trainees judge that they have been allowed to fail, their interpretation of the event is coloured by their attribution of supervisor intentions. Perceived intentions might impact the educational benefit of the experience. In order to support trainees' sense-making, we suggest that supervisory conversations during and after failure events should use more explicit language to discuss failures and explain supervisory intentions.

Which Entrustment-Supervision Scale is Right for Pharmacy Education?

Entrustable professional activities (EPAs) are an outgrowth of the competency-based educational model to support workplace-based learner assessments and evaluation. A learner's performance of EPAs is assessed by the degree of provided entrustment and required supervision rather than by a score, percentage, or letter grade typically assigned in traditional academic coursework. Entrustment-supervision (ES) scales are used to document learner progression and steer learner development over time. The purpose of this article is to critique various ES tools in health professions education for utilization within an EPA framework for learner assessment in workplace-based settings and to determine which will best suit pharmacy education. Exploring the advantages and disadvantages across all types of ES scales is a critical step in determining the most useful ES tool for use within a specific pharmacy institution and across the Academy. An ES scale with the traditional 5 levels, a prospective assessment frame, and increased stratification at lower levels should be recommended by the Academy and utilized in workplace-based settings for formative and summative assessment to provide more valid assessment of learners, support the ideal of life-long learning, and give more meaning for pharmacy faculty and learners within assessment.

Developing entrustable professional activities for family medicine training in South Africa

Workplace-based assessment (WPBA) is becoming part of high-stake assessments in specialist training. Entrustable professional activities (EPAs) are a recent addition to WPBA. This is the first South African publication on developing EPAs for postgraduate family medicine training. An EPA is a unit of practice, observable in the workplace, constituting several tasks with underlying knowledge, skills and professional behaviours. Entrustable professional activities allow for entrustable decisions regarding competence in a described work context. A national workgroup representing all nine postgraduate training programmes in South Africa has developed 19 EPAs. This new concept needs change management to understand the theory and practice of EPAs. Family medicine departments with large clinical workloads are small, necessitating navigating logistical issues to develop EPAs. It has unmasked existing workplace learning and assessment challenges.Contribution: This article contributes new thinking to developing EPAs for family medicine in an effort to understand more authentic WPBA nationally.

Entrust Me: Embedding Entrustable Professional Activities in a Gastroenterology Residency Program

Entrustable Professional Activities (EPAs) are defined as a process of gradually entrusting key tasks to specialty fellows during their training. EPAs are an important component of competency-based medical education; the concept of entrustment is also familiar and intuitive to clinical faculty, even inexperienced evaluators even if not termed as such. In this paper, we describe the process of how the authors adopted an established EPA framework for gastroenterology training, using EPAs to guide curriculum and faculty development and assessment in ten steps: (1) adopting an established framework, (2) mapping EPAs to relevant competencies, (3) specifying expected behaviors for competency of each EPA, (4) training faculty and fellows to have a shared mental model, (5) designing the training curriculum and educational strategies based on EPAs, (6) determining the assessment strategy, (7) designing the assessment tool, (8) ensuring clarity in how assessment data are used to make summative decisions, (9) changing feedback culture of fellows, and (10) using a longitudinal coaching system to improve EPA performance.

Entrustable Professional Activities and Their Relevance to Pediatric Cardiology Training

Entrustable professional activities (EPAs) have become a popular framework for medical trainee assessment and a supplemental component for milestone and competency assessment. EPAs were developed to facilitate assessment of competencies and furthermore to facilitate translation into clinical practice. In this review, we explore the rationale for the introduction of EPAs, examine whether they fulfill the promise expected of them, and contemplate further developments in their application with specific reference to training in pediatric cardiology.

Entrustable professional activities for residency in general internal medicine: a systematic review

CONTEXT

Entrustable Professional Activities (EPAs) are observable tasks that are regular parts of a physician's daily clinical work. Before being permitted to accomplish these tasks independently, trainees must gain their supervisors' trust. Defining the list of EPAs that should be mastered by the end of a residency is critical to setting clear expectations about autonomous practice.

OBJECTIVE

To collect all the lists of EPAs defined for residencies in general internal medicine and synthesise them into a reference work useful for developing new lists of EPAs or improving existing ones.

METHOD

This systematic review searched five databases and relevant grey literature using keywords related to EPAs and postgraduate education, from 2005, when the first article on EPAs was published, to April 2022. Inclusion criteria were the availability of an EPAs list and a focus on general internal medicine. Two reviewers independently selected the studies, extracted data and performed a quality assessment using QATSDD and AACODS tools. Mean values and inter-rater reliability were calculated.

RESULTS

The review yielded 3292 records, with 16 articles meeting the inclusion criteria, mostly from North America. Synthesising their 16 lists generated 395 EPAs. The reviewers then inductively categoried those EPAs, 308 of which fell into 6 domains, 14 themes and 24 subthemes. The domains were: (1) care and management of the general adult population (n = 103 EPAs); (2) care and management of patients with specific needs (n = 67); (3) care coordination and communication (n = 52); (4) management and leadership (N = 21); (5) healthcare quality, education, and research (n = 41); and (6) miscellaneous (n = 24). The remaining 87 EPAs were disease-specific and did not fit into this categorisation.

CONCLUSIONS

Categorising EPAs created a unique synthesis of the existing lists of EPAs for educating residents in general internal medicine. This synthesis could be used as a reference for anyone tasked with developing new lists of EPAs or improving existing ones.

Developing and Mapping Entrustable Professional Activities with Saudi Meds Competency Framework: A Consensus Study

PURPOSE

This study aimed at developing a national consensus on entrustable professional activities (EPAs) for Saudi undergraduate medical education and mapping them with the "Saudi Meds" competency framework.

METHODS

A three phased approach was used. Phase 1 consisted of identifying and developing EPAs; Phase 2 consisted of building a national consensus on developed EPAs (validation process); and Phase 3 consisted of mapping the validated EPAs with the Saudi Meds competency framework. Nominal group and modified Delphi techniques were used to develop consensus on EPAs. Classical test theory-based item analysis was conducted to establish validity and reliability of finalized EPAs.

RESULTS

Fifteen expert medical educationists and 109 academic leaders from 23 medical schools participated in the validation process. The study achieved a consensus on 10 core EPAs with an overall reliability (Cronbach's Alpha) of 0.814. The item-total correlation ranged from 0.341 to 0.642.

CONCLUSION

This study results in a national consensus on generic, comprehensive and region-specific EPAs that have been mapped with Saudi Meds competency framework. Our study is the first step in the direction of facilitating EPA-based curricular reforms in Saudi medical schools.

Assessment of final-year medical students' entrustable professional activities after education on an interprofessional training ward: A case-control study

INTRODUCTION

Interprofessional training wards (ITWs) are implemented to provide medical students with a holistic and authentic health care experience to improve their clinical competencies. Controlled outcome studies assessing students' competencies after ITW-training are uncommon. In this case-control study, we assessed final-year medical students who received ITW-training regarding entrustable professional activities (EPAs) and communicative as well as social competencies.

METHODS

In March 2021, 32 final-year students, 16 with (ITW group) and 16 without (control group) a previous four-week placement on an ITW participated in a training simulating the first day of residency. The simulated patients assessed students' communication and interpersonal skills for history taking with the ComCare index after every consultation. Twelve prospective EPAs were assessed by three senior physicians after watching videos of the students' case presentations.

RESULTS

While baseline characteristics and ComCare index ratings were not significantly different between the two groups, the overall mean entrustment level for the 12 EPAs was significantly higher (p < 0.001) in the ITW group compared to the control group (median = 3.15 versus 2.22). The interrater reliability for all EPAs was high and entrustment in students from the ITW group was significantly higher in 10 out of 12 EPAs.

DISCUSSION

ITW training seems to prepare medical students well to practice competencies which are relevant for prospective entrustment decisions and can be deduced by senior physicians from case presentations. Further studies with larger student cohorts are needed to corroborate this finding and observable EPAs could also be defined to assess students' competencies after ITW training.

How can Entrustable Professional Activities serve the quality of health care provision through licensing and certification?

This paper about Entrustable Professional Activities (EPAs) was solicited to support the discussion about the future of licensing within the Medical Council of Canada. EPAs, units of professional practice to be entrusted to learners or professionals once they have shown to possess sufficient competence, were proposed in 2005 to operationalize competency-based postgraduate medical education and have become widely popular for various health professions education programs in many countries. EPAs break the breadth of competence for license down to units of practice that can be overseen, assessed, monitored, documented, and entrusted. EPAs together may constitute an individual's portfolio of qualifications, and define a scope of practice. A medical license and a specialty certification can then be defined as the required combination of EPAs for which one is qualified at any specific moment in time. That 'snapshot' could change over time and reflect the professional development of the individual, both in their competence and in their privileges to practice. Micro-credentialing and digital badges might become an adequate option to show-case one's scope of practice at any time and operationalize the idea of a dynamic portfolio of EPAs.

A Pilot Study of the Generalizability of Preclinical Entrustment Assessments in Undergraduate Medical Education

PURPOSE

The reproducibility and consistency of assessments of entrustable professional activities (EPAs) in undergraduate medical education (UME) have been identified as potential areas of concern. EPAs were designed to facilitate workplace-based assessments by faculty with a shared mental model of a task who could observe a trainee complete the task multiple times. In UME, trainees are frequently assessed outside the workplace by faculty who only observe a task once.

METHOD

In November 2019, the authors conducted a generalizability study (G-study) to examine the impact of student, faculty, case, and faculty familiarity with the student on the reliability of 162 entrustment assessments completed in a preclerkship environment. Three faculty were recruited to evaluate 18 students completing 3 standardized patient (SP) cases. Faculty familiarity with each student was determined. Decision studies were also completed. Secondary analysis of the relationship between student performance and entrustment (scoring inference) compared average SP checklist scores and entrustment scores.

RESULTS

G-study analysis revealed that entrustment assessments struggled to achieve moderate reliability. The student accounted for 30.1% of the variance in entrustment scores with minimal influence from faculty and case, while the relationship between student and faculty accounted for 26.1% of the variance. G-study analysis also revealed a difference in generalizability between assessments by unfamiliar (φ = 0.75) and familiar (φ = 0.27) faculty. Subanalyses showed that entrustment assessments by familiar faculty were moderately correlated to average SP checklist scores (r = 0.44, P < .001), while those by unfamiliar faculty were weakly correlated (r = 0.16, P = .13).

CONCLUSIONS

While faculty and case had a limited impact on the generalizability of entrustment assessments made outside the workplace in UME, faculty who were familiar with a student's ability had a notable impact on generalizability and potentially on the scoring validity of entrustment assessments, which warrants further study.

A Literature Review: Entrustable Professional Activities, an assessment tool for postgraduate dental training?

Assessing when dental trainees are ready to independently undertake clinical procedures at specialist level is critical for dental postgraduate programmes to determine when a trainee is 'work ready', in addition to ensuring patient safety. Entrustable professional activities (EPA) are a novel method of competency-based assessment. An EPA is a unit of professional practice or critical clinical activity identified within dental training programmes, which should be assessed during training, to establish if trainees are ready for independent practice, with a progressive decrease in supervision, based on supervisors' entrustment decisions. This article describes EPAs, entrustment decisions, including entrustment supervision scales and the process recommended to develop EPAs within dental curricula. EPAs have not been formally introduced for assessment within dental education programmes in the United Kingdom, but recent developments have been described in undergraduate dental education globally. Clinical significance: Competency-based assessments need to be continually developed to adapt to rapidly changing population health care and dental needs, to determine when dental trainees are ready for independent clinical practice. Early development of entrustable professional activities for assessment in undergraduate dental programmes has been well received by both trainees and supervisors. Further investigation is required to consider formal development of EPAs within postgraduate dental programmes.

Exploring how feedback reflects entrustment decisions using artificial intelligence

CONTEXT

Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs.

METHODS

In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment-based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction.

RESULTS

We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently.

CONCLUSIONS

Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal-setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.

Trust and risk pitfalls in medical education: A qualitative study of clinical teachers

PURPOSE

Learning opportunities in teaching hospitals are gated by clinical teachers. One way to unpack their decision-making is to employ a 'trust and risk' model. This study aimed to uncover clinical teachers' experience of trust, risk and vulnerability as they participate in medical education.

METHODS

Hospital-based clinical teachers were interviewed about trust, risk and vulnerability in medical education. Data analysis was undertaken using a constructivist, qualitative framework.

RESULTS

Twenty demographically diverse clinical teachers participated. Trust and risk were regarded as fundamental workplace and teaching concepts. Their concerns fell into three domains of risk: clinical, teaching and personal. Being trusted unlocked clinical learning opportunities, whereas trust failure limited future participation. Feeling trusted or not affected wellbeing and self-efficacy. Trust and risk pitfalls in education included bias, asymmetry and sidelining.

CONCLUSIONS

This study adds to the literature by voicing clinical teachers' personal risks and vulnerabilities. Attention was drawn to the benefits of being perceived as trustworthy, and to the clinical, teaching and personal vulnerabilities of trust failure.If expert judgement of trustworthiness is to be legitimised as meaningful assessment, clinical teachers must be aware not only of how trust is built, but also the pitfalls of trust failure.

Supported Independence: The Role of Supervision to Help Trainees Manage Uncertainty

PURPOSE

Safe and effective supervised practice requires a negotiated partnership between trainees and their supervisors. Substantial work has explored how supervisors make judgments about trainees' readiness to safely engage in critical professional activities, yet less is known about how trainees leverage the support of supervisors when they perceive themselves to be at the limits of their abilities. The purpose of this study is to explore how trainees use supervisory support to navigate experiences of clinical uncertainty.

METHOD

Using a constructivist grounded theory approach, the authors explored how novice emergency medicine trainees conceptualized the role of their supervisors during experiences of clinical uncertainty. They employed a critical incident technique to elicit stories from participants immediately following clinical shifts between July and September 2020, and asked participants to describe their experiences of uncertainty within the context of supervised practice. Using constant comparison, 2 investigators coded line-by-line and organized these stories into focused codes. The relationships between these codes were discussed by the research team, and this enabled them to theorize about the relationships between the emergent themes.

RESULTS

Participants reported a strong desire for supported independence, where predictable and accessible supervisory structures enabled them to work semiautonomously through challenging clinical situations. They described a process of borrowing their supervisors' comfort during moments of uncertainty and mechanisms to strategically broadcast their evolving understanding of a situation to implicitly invoke (the right level of) support from their supervisors. They also highlighted challenges they faced when they felt insufficiently supported.

CONCLUSIONS

By borrowing comfort from-or deliberately projecting their thinking to-supervisors, trainees aimed to strike the appropriate balance between independence for the purposes of learning and support to ensure safety. Understanding these strategic efforts could help educators to better support trainees in their growth toward self-regulation.

The recommended description of an entrustable professional activity: AMEE Guide No. 140

Entrustable professional activities (EPAs) have received much attention in the literature since they were first proposed in 2005. Useful guidelines, workshops, courses, and conferences have supported faculty in developing programs and designing assessment procedures using EPAs and entrustment decision-making. Yet, the need for clarification remains, particularly as more programs make the step from design to implementation.Well-written EPAs provide a natural construct to establish the outcome of training. To be useful, EPAs require more than a suitable title. This AMEE Guide elaborates eight sections of a full EPA description, and provides explanations and justifications for each. These sections are: title; specification and limitations; risks in case of failure; most relevant competency domains; knowledge, skills, attitudes and experiences; information sources to assess progress and support summative entrustment; entrustment/supervision level expected at which stage of training; and time period to expiration if not practiced.Constructing fully elaborated EPAs creates a shared mental model amongst learners and programs, informs competency-based curriculum design, directs ad-hoc and formal entrustment decision-making, and provides standards for certifying bodies and boundaries for scope of practice. The framework intends to support curricular leaders looking to adopt new EPAs, or revise and define established EPAs for competency-based education.

Feasibility, and validity aspects of Entrustable Professional Activity (EPA)-based assessment in general practice training

INTRODUCTION

Entrustable Professional Activities (EPAs) are developed to support the practical application of competency frameworks in postgraduate medical education (PGME) programmes. EPAs are used for the assessment of the trainees' competence development, which takes place by means of an entrustment decision, aiming to stimulate learning and independent practice in trainees. In this pilot study, we explore the feasibility and validity of EPA-based assessment in a General Practice (GP) training programme.

METHODS

We used questionnaires to evaluate trainers' and trainees' experiences with the use of six EPAs for trainee learning, assessment and independent practice at the Out-of-Hours GP Center. Data were analysed quantitatively and qualitatively. Additionally, we examined the inter-item correlation between scores on EPA-based assessment and competency-based assessment using Spearman's Rho.

RESULTS

EPA-based assessment provided opportunities for giving concrete feedback and substantiating competency-based assessment. No consistent correlation between EPA-based assessment and competency-based assessment could be detected. Only later in the course of the training programme a correlation was found between the EPA scores and the degree of independence of trainees.

DISCUSSION

Results of this pilot study confirm the theories behind EPAs, as well as earlier research on EPAs in the workplace regarding trainee learning, assessment and independent practice. An important limitation of this study was the COVID-19 pandemic, as it influenced the results through reduced inclusion and follow-up, and through the impact on the workplace and trainee learning possibilities. Further research is needed to determine how EPAs support independent practice of trainees, as well as the assessment of trainee competency development.

The Implementation of Entrustable Professional Activities in Postgraduate Medical Education in the Netherlands: Rationale, Process, and Current Status

Postgraduate medical education in the Netherlands has adopted competency-based education since the turn of the century. In 2006, the CanMEDS competency framework was introduced. A 2013 government plan to reduce the length and budgets of training programs led the Dutch Association of Medical Specialists (DAMS) to respond with a proposal to create more flexibility and individualization rather than a blunt cut in the length across all training programs. DAMS launched a government-funded, nation-wide, 4-year project (2014-2018) to blueprint the reform of postgraduate medical education in this direction. To achieve competency-based individualization, the fixed duration of postgraduate programs was abandoned, and entrustable professional activities (EPAs) were introduced in all specialty programs. Implementation of this new generation of programs took place in 2017-2019 in all disciplines. The project focused on EPA-based individualization of all programs, while addressing issues of the continuity of patient care in time-variable programs and the legal and regulatory consequences of individualization. About 30 specialty programs were revised at national, regional, local, and individual levels to incorporate EPAs; portfolio systems were adapted, clinical competency committees were installed for all programs, and procedures for summative entrustment decision making were elaborated. This paper reports on the rationale and the process that led to a more time-variable postgraduate education landscape, and, on average, a shortening of training length by 3 months.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

Clarifying essential terminology in entrustment

With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

True Entrustment Decisions Regarding Entrustable Professional Activities Happen in the Workplace, not in the Classroom Setting

Entrustable Professional Activities (EPAs) are workplace responsibilities that directly impact patient care. The use of EPAs allows pharmacy faculty and preceptors to provide learners with feedback and assessment in the clinical setting. Because they focus assessment on a learner's execution of professional activities which requires integration of the respective competencies, EPAs help provide a more holistic picture of a learner's performance. Using EPAs to backwards design classroom learning for those competencies is highly encouraged, but instructors cannot or should not assess performance and make entrustment decisions using EPAs in the classroom setting for several reasons: a learner's classroom performance usually does not predict clinical performance very well, assessment of EPAs require direct observation of the learner performing the EPAs, EPA assessment requires multiple observations of the learner with different patients with varying level of acuity, and most importantly, EPA assessment must result in a decision to trust the learner to perform the clinical activity with limited supervision. By ensuring all entrustment decisions are made in a clinical or experiential setting, students will receive an accurate assessment and benchmark of their performance that will lead them one step closer to becoming independent practitioners.

Development of entrustable professional activities for a physiotherapy undergraduate programme in Singapore

BACKGROUND

The concept of entrustable professional activities (EPAs) as an assessment framework is new to undergraduate physiotherapy training in Singapore. The physiotherapy community was tasked to develop EPAs to reflect the core professional activities of an entry-level physiotherapist for its undergraduate programme.

METHODS

A qualitative participatory approach was used to develop entry-level EPAs for undergraduate physiotherapy students. Four phases occurred; (1) formation and training of the EPA writing workgroup from different sectors of the physiotherapy industry in Singapore; (2) conceptualization of the framework; (3) writing EPAs and; (4) iteration of EPAs through reflection, revision and consensus building.

RESULTS

Five core activities were identified for undergraduate physiotherapy students to perform upon graduation and were developed into entry-level EPAs. These entry-level EPAs were subsequently termed as core EPAs to reflect the core practices expected at entry level across specialties and sectors in Singapore. These EPAs were mapped to five competency domains and 12 competency descriptors. The sources of information to support entrustment decisions were (1) short practice observation, (2) entrustment-based discussion and (3) case notes evaluation. Remote supervision was selected as the level of entrustment that physiotherapy students must attain at graduation.

CONCLUSION

The development of EPAs utilized a ground-up and cross-sector approach to define entry-level EPAs as core physiotherapy activities expected to be performed and assessed during clinical training by undergraduate physiotherapy students. Future research is needed to validate the use of EPAs as an assessment framework for undergraduate physiotherapy education.

EPA-based assessment: Clinical teachers' challenges when transitioning to a prospective entrustment-supervision scale

BACKGROUND

This study explores the challenges clinical teachers face when first using a prospective entrustment-supervision (ES) scale in a curriculum based on Entrustable Professional Activities (EPAs). A prospective ES scale has the purpose to estimate at which level of supervision a student will be ready to perform an activity in subsequent encounters.

METHODS

We studied the transition to prospective assessment of medical students in clerkships via semi-structured interviews with twelve purposefully sampled clinical teachers, shortly after the introduction of a new undergraduate EPA-based curriculum and EPA-based assessment employing a prospective ES scale.

RESULTS

While some clinical teachers showed a correct interpretation, rating strategies also appeared to be affected by the target supervision level for completion of the clerkship. Instructions to estimate readiness for a supervision level in the future were not always understood. Further, teachers' interpretation of the scale anchors relied heavily on the phrasing.

DISCUSSION

Prospective assessment asks clinical teachers to make an extra inference step in their judgement process from reporting observed performance to estimating future level of supervision. This requires a change in mindset when coming from a retrospective, performance-oriented assessment method, i.e., reporting what was observed. Our findings suggest optimizing the ES-scale wordings and improving faculty development.

Entrustment Decision Making: Extending Miller's Pyramid

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid.