Twelve tips for the implementation of EPAs for assessment and entrustment decisions

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Feeling the responsibility: Exploring the emotional experiences of final-year medical students when carrying out clinical tasks

PURPOSE

The concept of Entrustable Professional Activities (EPA) is increasingly used to operationalize learning in the clinical workplace, yet little is known about the emotions of learners feeling the responsibility when carrying out professional tasks.

METHODS

We explored the emotional experiences of medical students in their final clerkship year when performing clinical tasks. We used an online reflective diary. Text entries were analysed using inductive-deductive content analysis with reference to the EPA framework and the control-value theory of achievement emotions.

RESULTS

Students described a wide range of emotions related to carrying out various clinical tasks. They reported positive-activating emotions, ranging from enjoyment to relaxation, and negative-deactivating emotions, ranging from anxiety to boredom. Emotions varied across individual students and were related to the characteristics of a task, an increasing level of autonomy, the students' perceived ability to perform a task and the level of supervision provided.

DISCUSSION

Emotions are widely present and impact on the workplace learning of medical students which is related to key elements of the EPA framework. Supervisors play a key role in eliciting positive-activating emotions and the motivation to learn by providing a level of supervision and guidance appropriate to the students' perceived ability to perform the task.

American Dental Education Association Compendium Entrustable Professional Activities Workgroup report

PURPOSE

Entrustable professional activities (EPAs) are discrete clinical tasks that can be evaluated to help define readiness for independent practice in the health professions and are intended to increase trust in the dental graduate. EPAs provide a framework that bridges competencies to clinical practice. This report describes the work of the American Dental Education Association (ADEA) Compendium EPA Workgroup to develop a list of EPAs for dental education and supportive resources, including specifications and a glossary.

METHODS

Preliminary work including literature and resource review, mapping of existing competencies, and review of other health professions' EPAs informed the development of our EPAs list. Workgroup members achieved consensus using a modified Delphi process. A Qualtrics survey using a validated rubric for the assessment of EPAs as described in peer-reviewed literature was used. Dental educators, including academic deans, were surveyed for feedback on the content and format of the EPAs.

RESULTS

Based on findings in the literature analysis of existing EPAs and competencies in health professions, a list of EPAs was developed along with a description of specifications. The EPA workgroup (nine members from multiple institutions) used the Delphi process in receiving feedback from various experts. A list of 11 core EPAs was vetted by dental educators including academic deans (n = ∼23), and the process of development was reviewed by EPAs experts outside dental education. A glossary was developed to align language.

CONCLUSION

These EPAs define the scope of dental practice. This report represents Phase 1 of the EPA framework development and vetting process. Future directions will include a broader vetting of the EPA list, faculty development, and national standardized technology that support this work to optimize implementation.

Motivations for Entrustable Professional Activity Assessment: Gaps Between Curriculum Theory and Resident Reality

Background Previous research demonstrates mixed reactions from residents toward competency-based medical education (CBME), and entrustable professional activities (EPAs) specifically. However, understanding what motivates residents to obtain EPAs may be vital to the longevity of CBME, given the emphasis on assessment for learning under this paradigm. Objective This study explored resident perspectives across 3 domains: motivation for obtaining EPAs, perceived importance of EPAs, and overall thoughts on CBME curriculum. Methods This was a sequential exploratory mixed-methods study involving 2 phases of data collection. Phase 1 was semi-structured interviews with residents enrolled in CBME at one Canadian institution from November 2019 to July 2020. Analyses included thematic and manifest content analysis. Phase 2 was an electronic close-ended survey to capture residents' primary motivation for requesting EPAs and importance of EPAs for learning. Survey data were analyzed descriptively. Results Of 120 eligible residents, 25 (21%) and 107 (89%) participated in the interview and survey, respectively. Program requirement was the dominant motivation for obtaining EPAs. There was variability in perceived importance of EPAs on learning. Increased resident workload, gaming the system to maximize EPA scores, and lack of shared ownership from preceptors were cited as critiques of the curriculum. Survey responses corroborated interview findings. Conclusions Although many residents recognize the value of EPAs, the majority are not intrinsically motivated to seek out assessment under the current CBME framework.

Designing the Future of Medical Education: The EPA Framework as a Catalyst to Inform Family Planning and Reproductive Health Fellowship Training Program in Ethiopia Medical Education: An Exploratory Sequential Mixed Method Study

Introduction

Entrustable Professional Activities (EPAs) are tasks or responsibilities within a specific field that can be given to a learner once they are competent to perform them independently. EPAs are being used in various specialty programs and serving as valuable tool to inform educational program. However, due to disparities in professional practice between different contexts, the automatic transfer of a set of core EPAs is not feasible. Hence, our study aims to develop an EPA framework to inform the Family Planning and Reproductive Health Fellowship Program in the local context of Ethiopia.

Methods

We employed an exploratory mixed-method design, which involved the collection of qualitative data using the Nominal Group Technique and quantitative data through a nationwide survey in all residency training institutions across the country. Qualitative data analysis involved several steps, including compiling a list of tasks, removing duplicate tasks, reviewing EPAs using criteria and an equal rubric tool. For quantitative data analysis, descriptive statistics, validity index analysis, and intra-class correlation coefficients, were used.

Results

Seven senior panelists were able to propose a total of 57 EPAs, with 17 remaining after qualitative data analysis. The panelist evaluated the relevance of each EPA in the second phase. As a result, 17 EPAs received a content validity index of >0.83, indicating satisfactory relevance. In the national survey, experts reached a high level of final agreement regarding the relevance and representativeness of all 17 EPAs (ICC = 0.815, 95% CI [0.0.756,0.865], p.0001).

Conclusion

The final set of 17 end-of-training EPAs is valid, acceptable and representative of the discipline, and they can be used as a framework to inform Family planning and Reproductive Health Fellowship Program in Ethiopian medical education once these core EPA statements are described in sufficient detail. This can contribute to raise the quality of training and hence the quality of patient care.

The learning curve for robotic-assisted transperineal MRI/US fusion-guided prostate biopsy

Transperineal fusion prostate biopsy has a considerable learning curve (LC). Robotic-assisted transperineal MRI/Ultrasound fusion-guided biopsy (RA-TP-FBx) may have an easier LC due to automatization. We aimed to assess the LC of RA-TP-FBx and analyze its most difficult steps. We prospectively analyzed cases randomized to a biopsy-naïve urology resident, the chief resident, and an expert urologist in RA-TP-FBx (controls). We also analyzed consecutive cases in the LC of the expert. The LC was defined by procedure time, PCa detection rate (including stratification by PI-RADS), entrustable professional activities (EPA) assessment scores, and the NASA task load index. We collectively performed 246 RA-TP-FBx with the Mona Lisa device. Procedure time for residents decreased steeply from maximum 53 min to minimum 10 min, while the mean procedure time for the expert was 9 min (range 17-5 min). PCa detection for PI-RADS-4 lesions was 57% for the naïve resident, 61% for the chief resident and 62% for the expert. There was also no difference in Pca detection for PI-RADS-4 lesions when comparing the first and second half of the experts' biopsies (p = 0.8). Maximum EPA score was registered after 22 cases. Workload steeply declined. Proficient RA-TP-FBx performance appears feasible after 22 cases regardless of previous experience.

Competency-Based Cultural Safety Training in Medical Education at La Sabana University, Colombia: A Roadmap of Curricular Modernization

Issue: Cultural safety enhances equitable communication between health care providers and cultural groups. Most documented cultural safety training initiatives focus on Indigenous populations from high-income countries, and nursing students, with little research activity reported from low- and middle-income countries. Several cultural safety training initiatives have been described, but a modern competency-based cultural safety curriculum is needed. Evidence: In this article, we present the Competency-Based Education and Entrustable Professional Activities frameworks of the Faculty of Medicine at La Sabana University in Colombia, and illustrate how this informed modernization of medical education. We describe our co-designed cultural safety training learning objectives and summarize how we explored its impact on medical education through mixed-methods research. Finally, we propose five cultural safety intended learning outcomes adapted to the updated curriculum, which is based on the Competency-Based Education model. Implications: This article presents five cultural safety intended learning outcomes for undergraduate medical education. These learning outcomes are based on Competency-Based Education and the Entrustable Professional Activities framework and can be used by faculties of medicine interested in including the cultural safety approach in their curriculum.

Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program

BACKGROUND

Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design.

METHODS

We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program's use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs.

RESULTS

Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a "numbers game" residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning.

CONCLUSION

Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training.

Preceptor perceptions of a redesigned entrustable professional activity (EPA) assessment tool in pharmacy practice experiences

INTRODUCTION

The purpose of this study was to determine pharmacy preceptor perceptions of an entrustable professional activity (EPA) evaluation tool redesigned in 2019 and launched for use in May 2020.

METHODS

Participants received an optional survey in June 2021 regarding their perceptions of the redesigned EPA assessment tool. The survey used a combination of open- and closed-ended questions regarding preceptor perceptions of the revised form, including perceptions of accuracy, usability, ease of use, grading, EPAs, and form layout. Preceptor perceptions were measured using seven-point Likert-type items and analyzed with descriptive statistics. Open-ended items were coded by at least two faculty members using qualitative content analysis.

RESULTS

We received 106 responses (19.5% response rate) and analyzed a total of 91 responses. Most questions related to perceived accuracy were rated positively, and preceptors indicated it was easier to assign grades and provide feedback on EPAs using the new form. Preceptors agreed that EPAs indicated when students on clinical rotations excel and when they need to improve. Preceptors reported they could effectively evaluate students and do so in a way that saves them time. Overall, preceptors agreed that the form instructions helped them accurately assess student performance, and the new narratives helped them remember the assessment standards.

CONCLUSIONS

Overarchingly, preceptors perceived that the new EPA assessment tool allowed them to provide feedback and grades more accurately, efficiently, effectively, and easily. This information will assist our institution in determining next best steps for EPA assessment tool updates.

Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison

Background

Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment.

Methods

EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members.

Results

Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively.

Conclusions

The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.

Entrustable professional activities in undergraduate dental education: A practical model for development and validation

OBJECTIVE

Entrustable professional activities (EPAs) are tasks that a person who is qualified or is in the process to be credentialed are allowed to engage. There are several levels of entrustment based on degrees of supervision assigned to each EPA. This paper aims to describe the process and outcome of creating EPAs; validate EPAs relevant to undergraduate dental training.

METHODS

A draft set of EPA statements was developed based on the consensus of an expert panel. These were then mapped to the nationally determined minimum experience thresholds (clinical and procedural experiences/competencies) and aligned to task-based instructional strategy. The EPAs were validated to improve the relevance by using a criterion-based rubric.

RESULTS

An end-to-end process workflow led to the development of an EPA-based educational framework to bridge the gaps in the curriculum. The process identified a total of 41 EPAs and out of which, 10 EPAs were notated as core EPAs and will be subjected to structured workplace-based assessment complying to the national standards. The validation exercise rated core EPAs with an overall score matching close to the cut-off of 4.07 (Equal rubric).

CONCLUSION

The end-to-end process workflow provided the opportunity to elaborate a structured process for the development of EPAs for undergraduate dental education. As validation is a continuous process, feedback from implementation will inform the next steps.

[Closer examination of specialization training in urology in German-speaking countries : Differences, similarities, advantages, disadvantages, and future perspectives]

BACKGROUND

Specialist titles are mutually recognized in the DACH (Germany [D], Austria [A], and Switzerland [CH]) region. During continuing education, however, there are significant differences in structure, organization and content.

OBJECTIVES

Organization, formalities and responsibilities, as well as admission requirements for the final examinations, in particular the examination formats in the DACH countries will be examined. In addition, these will be critically examined from the perspective of modern medical education.

MATERIALS AND METHODS

Documents publicly available on the Internet from the responsible authorities for continuing medical education in the DACH region were analyzed. In addition, a narrative review of the literature in medical databases on the topic of continuing medical education in urology was performed.

RESULTS

The minimum duration of residency is 5 (D) to 6 years (A, CH). Basic (surgical) training is mandatory in Austria and Switzerland. In Switzerland, the training site must be changed at least once. Part-time training is possible in all countries. Research and participation in annual congresses are mandatory only in Switzerland. Formative examinations are only required in Switzerland. In all countries, at least one summative examination is required to obtain the specialist title. In Austria and Switzerland, the written part of the European Board of Urology (EBU) examination must be passed.

CONCLUSION

From a medical education perspective, Switzerland currently offers and demands the most modern elements in specialist training. The certified structured continuing education modelled on the "Deutschen Gesellschaft für Urologie" (DGU; "Weiterbildungscurriculum Urologie", WECU) in Germany integrates modern approaches in Germany. With timely implementation of, for example, entrustable professional activities (EPAs), urology in German-speaking countries could be pioneering nations and disciplines internationally.

Final-year information on didactic and organizational issues for students and supervising physicians - project report on the development and implementation of the cross-site website PJ-input

OBJECTIVE

Final-year training is becoming increasingly important in medical studies and requires a high degree of personal responsibility from students. It is the task of supervising physicians to make informal learning opportunities available to students when working with and on patients and to gradually transfer responsibility to them. Both students and physicians have a great need for information regarding the contextual conditions and didactic realization of this transfer of responsibility. Up to now, the faculties have only provided information and support in a sporadic manner and with little standardization. With MERLIN, the joint project undertaken by the Competence Network for Teaching Medicine in Baden-Württemberg, a platform for the final year was developed and released on the web. The aim was to bundle information in order to support students and supervising physicians in their teaching-learning process and to improve the quality of teaching in the final year.

PROJECT DESCRIPTION

The development process of this platform took place in several steps across all faculties. Content and materials were compiled and structured based on a needs assessment. The first draft was evaluated by means of a simulation by students and then revised. A professional internet agency was involved for the technical implementation. The newly designed website PJ-input ("PJ" being the abbreviation for "Praktisches Jahr", the final year) contains areas for students and supervising physicians, as well as faculty-specific and general information about the final year. Faculty-specific content can be entered directly by the respective staff via an input mask and updated at any time. The provision of didactic materials can support competency-oriented teaching and learning in the final year. Here, for example, the concept of the Entrustable Professional Activities (EPA) was taken up, which gives students and supervising physicians orientation for the gradual assumption or transfer of responsibility. The platform was launched in spring 2021. Usage behavior is continuously recorded via the web application.

RESULTS AND CONCLUSION

The evaluation results show that the website is visited often and perceived as supportive. Increasing usage figures and the high frequency of use by students in the sections "im PJ" (during the final year) and "nach dem PJ" (after the final year) for the faculties involved in the MERLIN project confirm the target group-oriented design and use. The site should be promoted even more to pre-final-year students, as well as across state borders and to the target group of faculties. It is expected that nationwide faculty participation will make a significant contribution to the competency-based shift in teaching and the standardization of training during the final year of study under the new licensing regulations.

An Innovative Undergraduate Medical Curriculum Using Entrustable Professional Activities

The need to educate medical professionals in changing medical organizations has led to a revision of the Radboudumc's undergraduate medical curriculum. Entrustable professional activities (EPAs) were used as a learning tool to support participation and encourage feedback-seeking behavior, in order to offer students the best opportunities for growth. This paper describes the development of the Radboudumc's EPA-based Master's curriculum and how EPAs can facilitate continuity in learning in the clerkships. Four guiding principles were used to create a curriculum that offers possibilities for the students' development: (1) working with EPAs, (2) establishing entrustment, (3) providing continuity in learning, and (4) organizing smooth transitions. The new curriculum was designed with the implementation of EPAs and an e-portfolio, based on these 4 principles. The authors found that the revised curriculum corresponds to daily practice in clerkships. Students used their e-portfolios throughout all clerkships, which stimulates feedback-seeking behavior. Moreover, EPAs promote continuity in learning while rotating clerkships every 1 to 2 months. This might encourage curriculum developers to use EPAs when aiming for greater continuity in the development of students. Future research needs to focus on the effect of EPAs on transitions across clerkships in order to further improve the undergraduate medical curriculum.

Assessment of final-year medical students' entrustable professional activities after education on an interprofessional training ward: A case-control study

INTRODUCTION

Interprofessional training wards (ITWs) are implemented to provide medical students with a holistic and authentic health care experience to improve their clinical competencies. Controlled outcome studies assessing students' competencies after ITW-training are uncommon. In this case-control study, we assessed final-year medical students who received ITW-training regarding entrustable professional activities (EPAs) and communicative as well as social competencies.

METHODS

In March 2021, 32 final-year students, 16 with (ITW group) and 16 without (control group) a previous four-week placement on an ITW participated in a training simulating the first day of residency. The simulated patients assessed students' communication and interpersonal skills for history taking with the ComCare index after every consultation. Twelve prospective EPAs were assessed by three senior physicians after watching videos of the students' case presentations.

RESULTS

While baseline characteristics and ComCare index ratings were not significantly different between the two groups, the overall mean entrustment level for the 12 EPAs was significantly higher (p < 0.001) in the ITW group compared to the control group (median = 3.15 versus 2.22). The interrater reliability for all EPAs was high and entrustment in students from the ITW group was significantly higher in 10 out of 12 EPAs.

DISCUSSION

ITW training seems to prepare medical students well to practice competencies which are relevant for prospective entrustment decisions and can be deduced by senior physicians from case presentations. Further studies with larger student cohorts are needed to corroborate this finding and observable EPAs could also be defined to assess students' competencies after ITW training.

"The Most Crushing Thing": Understanding Resident Assessment Burden in a Competency-Based Curriculum

BACKGROUND

Competency-based medical education (CBME) was expected to increase the workload of assessment for graduate training programs to support the development of competence. Learning conditions were anticipated to improve through the provision of tailored learning experiences and more frequent, low-stakes assessments. Canada has adopted an approach to CBME called Competence by Design (CBD). However, in the process of implementation, learner anxiety and assessment burden have increased unexpectedly. To mitigate this unintended consequence, we need a stronger understanding of how resident assessment burdens emerge and function.

OBJECTIVE

This study investigates contextual factors leading to assessment burden on residents within the framework of CBD.

METHODS

Residents were interviewed about their experiences of assessment using constructivist grounded theory. Participants (n=21) were a purposive sample from operative and perioperative training programs, recruited from 6 Canadian medical schools between 2019 and 2020. Self-determination theory was used as a sensitizing concept to categorize findings on types of assessment burden.

RESULTS

Nine assessment burdens were identified and organized by threats to psychological needs for autonomy, relatedness, and competence. Burdens included: missed opportunities for self-regulated learning, lack of situational control, comparative assessment, lack of trust, constraints on time and resources, disconnects between teachers and learners, lack of clarity, unrealistic expectations, and limitations of assessment forms for providing meaningful feedback.

CONCLUSIONS

This study contributes a contextual understanding of how assessment burdens emerged as unmet psychological needs for autonomy, relatedness, and competence, with unintended consequences for learner well-being and intrinsic motivation.

Making prospective entrustment decisions: Knowing limits, seeking help and defaulting

INTRODUCTION

Many studies focused on entrustment have not considered prospective entrustment decisions, where clinical competency and entrustment committees determine how much supervision trainees will require in the future for tasks occurring in not completely known contexts. The authors sought to explore factors that influence and determine prospective entrustment decisions made by members of such committees in graduate medical education (GME) and undergraduate medical education (UME).

METHODS

The authors conducted a constructivist grounded theory study with 23 faculty participants from GME and UME clinical competency and entrustment committees in the United States between October 2020 and March 2021. Interviews sought to explore factors and considerations participants weigh in making prospective entrustment decisions about trainees. Data collection and analysis occurred in an iterative fashion, ensuring constant comparison. Theoretical sampling was used to confirm, disconfirm and elaborate on the evolving results.

RESULTS

Trainees' ability to know limits and seek help is the foundation of participants' prospective entrustment decision making. Most participants, however, describe a presumption of trainee readiness to progress and describe commonly making default prospective entrustment decisions unless 'red flags' in performance are present. Although participants desire sufficient and trusted data about trainee performance to inform decisions, they often lack it. Finally, the perceived permanence and consequences of prospective entrustment decisions influence how participants weigh other factors contributing to prospective entrustment decisions.

CONCLUSION

Trainees' ability to know limits and seek help appears to be the foundation of prospective entrustment decision making. Training programmes should strive to collect and employ robust data supporting and questioning the presence of these attributes.

Entrustable Professional Activities in Clinical Education: A Practical Approach for Advanced Nursing Education

BACKGROUND

Competency attainment is foremost in the ethos of this project that provides a practical guide to implementing entrustable professional activities (EPAs) as one approach to competency acquisition in the clinical education component of a health care program. EPAs are units of work that provide evidence of required competencies for the clinical education component of a program of study for educators, preceptors, and students.

PROBLEM

The American Association of Colleges of Nursing has raised the importance of competency-based education and need for a practical approach to assess clinical competency challenges (eg, EPAs) in nursing education.

APPROACH

The implementation of EPAs in the clinical component of advanced nursing education is detailed as an example in this article. Prioritizing a systematic approach, the Knowledge to Action framework, was chosen to guide the process.

CONCLUSION

Nursing must act now to put in place a robust competency acquisition and validation system.

Subjective achievement from psychiatry rotation in the Japanese postgraduate residency system: a longitudinal questionnaire study

BACKGROUND

Psychiatry rotation has been mandatory in the Japanese postgraduate residency system since 2020. Some psychiatry-related competency items are stipulated as mandatory for residents. The current study aimed to clarify whether psychiatry rotation affected residents' subjective achievement of these competency items.

METHODS

This longitudinal study was conducted among postgraduate residents who completed a rotation in the psychiatry department at Nagasaki University Hospital across two academic years (2020-2021). The survey was administered at the start and at the end of the psychiatry rotation. Residents evaluated their subjective understanding and confidence regarding initiating treatment for these competency items using a six-point Likert scale. The average scores for each item were compared between pre-rotation and post-rotation.

RESULTS

In total, 99 residents (91.7%) responded to this survey. Residents had significantly higher scores at post-rotation compared with pre-rotation in all psychiatry-related competency items in both subjective understanding and confidence in initiating treatment. Additionally, strong effect sizes were found for many items.

CONCLUSION

Residents improved learning about psychiatry-related competency items through psychiatry rotation. This finding suggests that it is reasonable for psychiatry rotation to be mandatory in the current Japanese postgraduate residency system. The importance of psychiatry is likely to increase in both undergraduate and postgraduate medical education in the future. It is necessary to continuously update educational strategies to meet changing social needs over time. As this study was conducted at a single institution, a multi-center study is needed to expand the current findings.

Stakeholders' perception on the implementation of Developmental Progress Assessment: using the Theoretical Domains Framework to document behavioral determinants

BACKGROUND

The widespread implementation of longitudinal assessment (LA) to document trainees' progression to independent practice rests more on speculative rather than evidence-based benefits. We aimed to document stakeholders' knowledge of- and attitudes towards LA, and identify how the supports and barriers can help or hinder the uptake and sustainable use of LA.

METHODS

We interviewed representatives from four stakeholder groups involved in LA. The interview protocols were based on the Theoretical Domains Framework (TDF), which contains a total of 14 behaviour change determinants. Two team members coded the interviews deductively to the TDF, with a third resolving differences in coding. The qualitative data analysis was completed with iterative consultations and discussions with team members until consensus was achieved. Saliency analysis was used to identify dominant domains.

RESULTS

Forty-one individuals participated in the study. Three dominant domains were identified. Participants perceive that LA has more positive than negative consequences and requires substantial ressources. All the elements and characteristics of LA are present in our data, with differences between stakeholders.

CONCLUSION

Going forward, we could develop and implement tailored and theory driven interventions to promote a shared understanding of LA, and maintain potential positive outcomes while reducing negative ones. Furthermore, ressources to support LA implementation need to be addressed to facilitate its uptake.

Design and implementation of a core EPA-based acting internship curriculum

PURPOSE

The Association of American Medical Colleges published Core Entrustable Professional Activities (Core EPAs), expected independent clinical skills for first-day interns. We sought to determine whether a required acting internship (AI) in the fourth-year curriculum could be used for summative assessment of students' mastery of Core EPAs to a predefined level that would readily generalize across disciplines and campuses.

METHODS

The University of North Dakota School of Medicine and Health Sciences MD Program created a standardized, required Core EPA-based AI curriculum for multiple specialties at multiple geographic sites providing a final entrustability assessment for 10 EPAs in a single course.

RESULTS

The course was successfully designed and launched for all students in a single class. During the AI, students functioned at the level of an acting intern, rated the courses as superior, and performed at satisfactory exit-level competence for 10 Core EPAs.

CONCLUSIONS

A standardized, EPA-based AI curriculum can provide an opportunity for exit level EPA assessment in the medical curriculum. This model functions well within multiple specialties and at diverse community-based, volunteer faculty teaching sites.

TrustEd: A Tool for Developing Intraoperative Entrustment Skills

OBJECTIVE

Toolkits to assess progressive resident autonomy are integral to the movement toward competency-based surgical education. OpTrust is one such tool validated for intraoperative assessment of both faculty and resident entrustment behaviors. We developed a supplementary tool to OpTrust that would aid faculty and residents in making meaningful improvements in entrustment behavior by providing talking points and reflection items tailored to different motivational styles as defined by Regulatory Focus Theory (RFT).

DESIGN

Existing literature about surgical entrustment was used to build a list of sample dialogue and self-reflection items to use in the operating room. This list was distributed as a survey to individuals familiar with OpTrust and RFT, asking them to categorize each item as Promotion-oriented, Prevention-oriented, or Either. The respondents then met to discuss survey items that did not reach a consensus until the group agreed on their categorization.

SETTING

University of Wisconsin, School of Medicine and Public Health, Madison, Wisconsin Michigan Medicine, Ann Arbor, Michigan PARTICIPANTS: Clinician and education researchers familiar with intraoperative entrustment and RFT RESULTS: Eight respondents completed the survey categorizing the talking points and reflection items by RFT (100% response rate). Six of these respondents attended the additional meeting to discuss discordant items. The input from this panel was used to develop "TrustEd," the supplementary tool that faculty and residents can quickly reference before beginning a case.

CONCLUSION

Although tools such as OpTrust allow intraoperative entrustment behaviors to be quantified, TrustEd offers concrete strategies for faculty and residents who are interested in improving those behaviors over time. Further study is needed to assess whether the use of TrustEd does in fact lead to durable behavior change and improvement in OpTrust scores.

Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents

BACKGROUND

A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency.

STUDY DESIGN

This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings.

RESULTS

From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05).

CONCLUSIONS

The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.

Development and consensus of entrustable professional activities for final-year medical students in anaesthesiology

Background

The transfer of classic concepts of competency-based medical education into clinical practice has been proven to be difficult in the past, being described as partially fragmented, misleading and inadequate. At the beginning of training, novice doctors commonly feel overwhelmed, overloaded and exposed to extreme time pressure. The discrepancy between expected and actual clinical competence of doctors at the start of their speciality training jeopardizes patient safety. The framework of Entrustable Professional Activities (EPAs) is a promising instrument to effectively integrate competency-based training into clinical practice and may help to close this gap and consequently to improve patient safety.

Methods

For anaesthesiology, we developed 5 EPAs for final-year medical students. The EPAs comprised the following seven categories: 1. Title, 2. Specifications, 3. Limitations, 4. Competency domains, 5. Knowledge, abilities and skills, professional attitudes, 6. Assessment and 7. Entrustment. Based on a modified, online-based Delphi study, we further developed and refined these EPAs. Education experts were recruited from the alumni network of the Master of Medical Education (MME) degree course from the University of Heidelberg, Germany.

Results

28 data sets were evaluated in three Delphi rounds. 82% of study participants had previous experience with EPAs. Qualitative and quantitative data formed the basis during the iterative process and resulted in complete descriptions of 5 EPAs for final-year medical students in anaesthesiology.

Conclusions

Our study including the associated description of 5 EPAs represent a further step and starting point for EPA-based curricula in medical training in Germany linking undergraduate training, to residency training and continuous medical education.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12871-022-01668-8.

An innovative interprofessional dental clinical learning environment using entrustable professional activities

INTRODUCTION

Changes in society, new public demands for dental care and contemporary educational insights have influenced dental education worldwide and demand a renewed perspective. Following this perspective, an innovative interprofessional dental Master curriculum was developed at Radboud University Nijmegen in the Netherlands.

EDUCATIONAL PRINCIPLES

The curriculum is based on five contemporary educational principles and the core of the curriculum consists of a Student Run Dental Clinic that is fully run by students under professional supervision.

THE NEW CURRICULUM

In the Student Run Dental Clinic, Master dental students and Bachelor oral hygiene students are responsible for the care of approximately 750 patients. The students work within the same clinic for 3 years and patients receive oral health care from the same students over a long period. The clinic is a longitudinal cross-disciplinary clinic with different dental subdisciplines. Entrustable professional activities (also known as EPAs), to our knowledge not yet widely used in dental education, were introduced to facilitate learning and assessment. Fourteen EPAs have been developed to stimulate interprofessional education and learning. Of these, five EPAs are identical for the dental and oral hygiene curriculum, leading to extended interprofessional education and learning in the Student Run Dental Clinic.

DISCUSSION

Preliminary results show that EPAs are generally well received by supervisors and students.

CONCLUSION

To monitor and investigate the exact effect of the interventions and underlying mechanisms, a research programme on interprofessional learning, practice-based learning and EPAs and entrustment in dental education was recently set up.

Analysis of Service Quality and Resident Supervision in Accordance with Hospital Accreditation Standards

Background Resident supervision plays an important role in accomplishing service excellence in teaching hospitals, while still supporting residents’ competency achievement. This preliminary study aims to obtain an overview of resident supervision and the achievement of quality services to conduct hospital accreditation standard.   Material and methods A retrospective study with a cross-sectional design was conducted in May 2020 involving Obstetrics and Gynecology residents who carried out a Caesarean section.  Data was collected from medical records manually, with additional information from Obstetrics and Gynaecology Department. Univariate analysis was completed to generate a profile of supervision and the achievement of quality services in accordance with  hospital accreditation standard.   Result Of the 86 Caesarean sections conducted in May 2020, only 75 involved residents, with 42 (56%) procedures carried out by chief residents and 33 (44%) by apprentice stage residents. Supervision during procedures was carried out in 74 cases (98.6%), with 33.3% given moderate-high supervision during hospitalization, 52% moderate low, and 14.7% low supervision. The completeness of the medical records revealed 85,3%  appropriate marked site, 96% surgery reports, 78,7% education sheet.  It is found that  64 % of pre-operative visits and  65,3% of post-operative visits were carried out by residents. In addition, case discussion, death case reports,  as well as  resident evaluation were carried out to conduct patient safety and quality service improvement.       Conclusion Supervisory evaluation could be provided directly or by conducting case discussions, as well as medical record review; otherwise, to enhance quality achievement, the development of an integrated supervision system is required.

Entrustable Professional Activities for Chinese Standardized Residency Training in Pediatric Intensive Care Medicine

BACKGROUND

Entrustable professional activities (EPAs) were first introduced by Olle ten Cate in 2005. Since then, hundreds of applications in medical research have been reported worldwide. However, few studies discuss the use of EPAs for residency training in pediatric intensive care medicine. We conducted a pilot study of EPA for pediatric intensive care medicine to evaluate the use of EPAs in this subspecialty.

MATERIALS AND METHODS

A cross-sectional study was implemented in pediatric intensive care medicine standardized residency training at the Qilu Hospital of Shandong University. An electronic survey assessing EPA performance using eight scales composed of 15 categories were distributed among residents and directors.

RESULTS

A total of 217 director-assessment and 44 residents' self-assessment questionnaires were collected, both demonstrating a rising trend in scores across postgraduate years. There were significant differences in PGY1-vs.-PGY2 and PGY1-vs.-PGY3 director-assessment scores, while there were no differences in PGY2-vs.-PGY3 scores. PGY had a significant effect on the score of each EPA, while position significantly affected the scores of all EPAs except for EPA1 (Admit a patient) and EPA2 (Select and interpret auxiliary examinations). Gender only significantly affected the scores of EPA6 (Report a case), EPA12 (Perform health education), and EPA13 (Inform bad news).

CONCLUSION

This study indicates that EPA assessments have a certain discriminating capability among different PGYs in Chinese standardized residency training in pediatric intensive care medicine. Postgraduate year, gender, and resident position affected EPA scores to a certain extent. Given the inconsistency between resident-assessed and director-assessed scores, an improved feedback program is needed in the future.

Design, implementation and evaluation of novel work-based clinical assessment tool: An e-portfolio with embedded Entrustable Professional Activities

BACKGROUND

A new concept in work-based assessment, Entrustable Professional Activities (EPAs) describe key activities a work-ready practitioner can perform independently. EPAs are widely used in medicine to support competency-based education and are gaining acceptance in other professions such as nursing, pharmacy and dietetics. There is currently no widely accepted work-based assessment tool for student clinical placement assessment in dietetics and EPAs offer potential to address this gap.

OBJECTIVES

To design, implement and evaluate a work-based assessment tool using EPAs embedded in an e-portfolio for use in clinical dietetics.

SETTING

An accredited dietetic training program in Australian university and affiliated metropolitan and rural hospitals.

PARTICIPANTS

Three consecutive cohorts of final year dietetic students (n = 126) and their professional placement supervisors (n = 101).

METHODS

A working party consisting of dietetic academics with experience in clinical dietetics and education developed clinical dietetic EPAs and milestones that were mapped to the National Competency Standards for Dietitians in Australia. A design model incorporating multiple rounds of user feedback was used to create an e-portfolio with the EPAs embedded. Students and supervisors were invited to complete anonymous on-line surveys with Likert responses (1 = highly dissatisfied, 5 = highly satisfied) to evaluate the face validity, acceptability and feasibility of the tool.

RESULTS

A total of 37 EPAs were developed with an accompanying four-point entrustment scale to measure performance. Evaluation survey response rates were high and ranged from 35 to 45% for students and 57-83% for supervisors across the study period. Both students and supervisors evaluated the tool positively with a mean overall satisfaction of 4.12 ± 0.69 and 4.03 ± 0.68, respectively. Survey items evaluating face validity, feasibility and acceptability scored >3.75 for both groups of users.

CONCLUSIONS

The EPA based e-portfolio demonstrated face validity, feasibility and acceptability and offers potential for a widely accepted tool for work-based assessment of clinical dietetic students.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Development of a veterinary emergency open standard competency framework using a competency-based model of medical education

OBJECTIVE

To address the shortage of emergency veterinarians, the profession is exploring accelerated training pathways. We sought to contribute to the solution by developing the foundation for an open standard, competency-based veterinary emergency training curriculum for use by any program. We also developed a curricular delivery, tracking, and assessment system to demonstrate how the framework can be integrated into training programs.  DESIGN: Hybrid Delphi method.  SETTING: Academia and referral practice.  ANIMALS: None.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

An emergency veterinary competency framework was developed by adapting the human Model of the Clinical Practice of Emergency Medicine, which aligns with the Competency-Based Veterinary Education framework, to produce 4 areas of core competency: Patient Care, Interpersonal/Communication, Professionalism, and Practice-based Learning/Improvement. A comprehensive list of veterinary emergency skills was generated and organized within the framework utilizing the hybrid Delphi method. An initial survey completed by 133 emergency and critical care specialists and emergency room clinicians produced data regarding the value of specific skills. An 11-member focus group consisting of survey participants iterated upon the survey results to produce a master library of skills and cases, including 56 Patient Care, 43 Interpersonal/Communication, 11 Practice-based Learning/Improvement, and 20 Professionalism skills, as well as 155 case types. The curricular delivery system tracks and assesses case management proficiency and development of knowledge and professional skills using a patient care eLearning program and simulation training environment.  CONCLUSIONS: The increasing need for emergency veterinarians is a shared industry-wide challenge. To contribute toward a collective solution, we have undergone an evidence-based process to create the foundation for an open standard competency framework composed of a library of skills and cases. We offer this open standard framework to the veterinary profession and hope it continues to grow and evolve as we drive toward developing competency-based training programs that address the shortage of emergency veterinarians.

Entrustable Professional Activities for Small-Group Facilitation: A Validation Study Using Modified Delphi Technique

:

Entrustable professional activities (EPAs) provide a novel approach to support teachers' structured professionalization and to assess improvement in teaching competence thereafter. Despite their novelty, it is important to assess EPAs as a construct to ensure that they accurately reflect the work of the targeted profession.

BACKGROUND:

The co-creation of an EPA framework for training and entrustment of small-group facilitators has been discussed in the literature. Although a rigorous design process was used to develop the framework, its content validity has not been established yet.Approach: A modified Delphi technique was used. Three survey rounds were conducted from December 2019 to April 2020. Expert health professions educationalists were recruited using purposive sampling and snowball techniques. In Round 1, a rubric consisting of seven items was used to assess the quality of nine pre-designed EPAs. In Round 2, competencies required to perform the agreed-upon EPAs were selected from 12 competencies provided. In Round 3, consensus was sought on sub-activities recommended for agreed-upon EPAs. Quantitative data were analyzed using multiple statistical analyses, including item-wise and rubric-wise content validity indices, asymmetric confidence interval, mean, standard deviation, and response frequencies. Qualitative data were thematically analyzed using content analysis.

FINDINGS:

Three of the nine proposed EPAs achieved statistical consensus for retention. These EPAs were: (1) preparing an activity, (2) facilitating a small-group session, and (3) reflecting upon self and the session. Nine of the 12 pre-determined competencies achieved consensus and were then mapped against each agreed-upon EPA based on their relevance. Finally, results indicated consensus on five, six, and four sub-activities for EPA 1, EPA 2, and EPA 3, respectively.

CONCLUSIONS:

The final framework delineates three EPAs for small-group facilitation and their associated sub-activities. The full description of each EPA provided in this article includes the title, context, task specification, required competencies, and entrustment resources. Program developers, administrative bodies, and teaching staff may find this EPA framework useful to structure faculty development, to entrust teachers, and to support personal development.

Feasibility of an Entrustable Professional Activity for Pathology Resident Frozen Section Training

Entrustable professional activities are an intuitive form of workplace-based assessment that can support competency-based medical education. Many entrustable professional activities have been written and published, but few studies describe the feasibility or implementation of entrustable professional activities in graduate medical education. The frozen section entrustable professional activit was introduced into the pathology residency training at the University of Vermont for postgraduate year 1 at the start of their training in frozen section. The feasibility of the entrustable professional activit was evaluated based on 3 criteria: (a) utilization, (b) support of frozen section training, and (c) generating data to support entrustment decision about residents’ readiness to take call. The entrustable professional activit was well utilized and satisfactory to residents, faculty, pathologists’ assistants, and Clinical Competency Committee members. Most members of the Clinical Competency Committee agreed they had sufficient data and noted higher confidence in assessing resident readiness to take call with the addition of entrustable professional activit to the residents’ assessment portfolio. Residents did not endorse it helped them prepare for call; however, the interruption to frozen section training due to the COVID-19 pandemic was a significant contributing factor. The frozen section entrustable professional activit is a feasible addition to pathology resident training based on utilization, support of training, and generation of data to support entrustment decisions for graduated responsibilities. The implementation and integration of the entrustable professional activit into pathology training at our institution is described with discussion of adjustments for future use.

Feasibility, and validity aspects of Entrustable Professional Activity (EPA)-based assessment in general practice training

INTRODUCTION

Entrustable Professional Activities (EPAs) are developed to support the practical application of competency frameworks in postgraduate medical education (PGME) programmes. EPAs are used for the assessment of the trainees' competence development, which takes place by means of an entrustment decision, aiming to stimulate learning and independent practice in trainees. In this pilot study, we explore the feasibility and validity of EPA-based assessment in a General Practice (GP) training programme.

METHODS

We used questionnaires to evaluate trainers' and trainees' experiences with the use of six EPAs for trainee learning, assessment and independent practice at the Out-of-Hours GP Center. Data were analysed quantitatively and qualitatively. Additionally, we examined the inter-item correlation between scores on EPA-based assessment and competency-based assessment using Spearman's Rho.

RESULTS

EPA-based assessment provided opportunities for giving concrete feedback and substantiating competency-based assessment. No consistent correlation between EPA-based assessment and competency-based assessment could be detected. Only later in the course of the training programme a correlation was found between the EPA scores and the degree of independence of trainees.

DISCUSSION

Results of this pilot study confirm the theories behind EPAs, as well as earlier research on EPAs in the workplace regarding trainee learning, assessment and independent practice. An important limitation of this study was the COVID-19 pandemic, as it influenced the results through reduced inclusion and follow-up, and through the impact on the workplace and trainee learning possibilities. Further research is needed to determine how EPAs support independent practice of trainees, as well as the assessment of trainee competency development.

OpTrust: An Effective Educational Bundle for Enhancing Faculty-resident Intraoperative Entrustment

OBJECTIVE

The purpose of this study was to measure the efficacy of a novel faculty and resident educational bundle focused on development of faculty-resident behaviors and entrustment in the operating room.

SUMMARY BACKGROUND DATA

As surgical training environments are orienting to entrustable professional activities (EPAs), successful transitions to this model will require significant faculty and resident development. Identifying an effective educational initiative which prepares faculty and residents for optimizing assessment, teaching, learning, and interacting in this model is critical.

METHODS

From September 2015 to June 2017, an experimental study was conducted in the Department of Surgery at the University of Michigan Health System (UMHS). Case observations took place across general, plastic, thoracic, and vascular surgical specialties. A total of 117 operating room observations were conducted during Phase I of the study and 108 operating room observations were conducted during Phase II following the educational intervention. Entrustment behaviors were rated for 56 faculty and 73 resident participants using OpTrust, a validated intraoperative entrustment instrument.

RESULTS

Multiple regression analysis showed a significant increase in faculty entrustment (Phase I = 2.32 vs Phase II = 2.56, P < 0.027) and resident entrustability (Phase I = 2.16 vs Phase II = 2.40, P < 0.029) scores following exposure to the educational intervention.

CONCLUSIONS

Our study shows improved intraoperative entrustment following implementation of faculty and resident development, indicating the efficacy of this innovative educational bundle. This represents a crucial component in the implementation of a competency-based assessment framework like EPAs.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment

INTRODUCTION

The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated.

METHODS

In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding.

RESULTS

Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout.

CONCLUSIONS

Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Using the Five Core Components of Competency-Based Medical Education to Support Implementation of CBVE

In recent years, veterinary education has begun the transition to competency-based models, recognizing that, like medical education, our goals include improved patient and client outcomes and the importance of learner-centered methods in education. Given that implementation of competency-based veterinary education (CBVE) is still in its relative infancy across many veterinary programs, we stand to gain from a unified approach to its implementation. As a guideline, the five core components of competency-based medical education (CBME) should serve to ensure and maintain fidelity of the original design of outcomes-based education during implementation of CBVE. Identified the essential and indispensable elements of CBME which include 1) clearly articulated outcome competencies required for practice, 2) sequenced progression of competencies and their developmental markers, 3) tailored learning experiences that facilitate the acquisition of competencies, 4) competency-focused instruction that promotes the acquisition of competencies, and 5) programmatic assessment. This review advocates the adoption of the principles contained in the five core components of CBME, outlines the approach to implementation of CBVE based upon the five core components, and addresses the key differences between veterinary and medical education which may serve as challenges to ensuring fidelity of CBVE during implementation.

Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

Entrustable professional activities framework for assessment in predoctoral dental education, developed using a modified Delphi process

PURPOSE/OBJECTIVES

The purpose of this study was to define and develop a set of Entrustable Professional Activities (EPAs) for dental education using a modified Delphi consensus approach. EPAs define the core tasks that a graduating dentist needs to perform independently in practice. The EPA framework facilitates assessment of competencies as they manifest in the tasks and independence needed to be ready for practice.

METHODS

Feedback was obtained from participants about a list of EPAs, with modifications made after each of the 3 rounds, using a modified Delphi approach. Phase 1 included attendees at the ADEA Fall 2017 meeting (n = 35) who participated in an EPA workshop primarily composed of academic deans. The Phase 2 "reactor panel" consisted of 10 dental schools' academic deans and other individuals with expertise and interest in dental curriculum and assessment (n = 31). Phase 3 participants were attendees at the ADEA CCI 2019 meeting (n = 91) who also participated in a 2-day EPA workshop.

RESULTS

In phase 1, overall ratings for acceptability of the EPAs were satisfactory. In phase 2, the next iteration of EPAs was judged as satisfactory for inclusion in curriculum, match well with clinical practice and clarity. In phase 3, the EPAs were judged as satisfactory for being an "entrustable, essential, and important task of the profession." Qualitative feedback suggested wording, measurability, and specific focus of EPA statements is important.

CONCLUSIONS

A preliminary set of EPAs was designed for predoctoral dental education through a systematic, careful consensus building approach involving a diverse set of participants.

Continuity of supervision: Does it mean what we think it means?

CONTEXT

Continuity of supervision (CoS) is generally accepted as an important element of competency-based medical education (CBME). However, collecting and interpreting evidence for its effectiveness are a challenge because we lack a shared understanding of CoS. Translating the available evidence about CoS into practice is an even greater challenge because the evidence largely exists in the undergraduate medical education (UME) literature, whereas literature about CBME is mostly situated in postgraduate medical education (PGME).

PROPOSAL

We explore the potential dangers of basing assumptions of the importance of CoS in CBME on evidence from the UME level where CBME is yet to be widely implemented. First, we discuss current understandings of what is meant by CoS and examine some of its evidence and where such evidence comes from. Next, we consider relevant theories related to CoS in the context of CBME and review how it is conceptualised in different educational models. We then discuss some contextual and pedagogical differences between UME and PGME when CoS is considered. Finally, we propose a shared understanding of CoS and outline implications and next steps to determine if the benefits of CoS seen at the UME level will also manifest with PGME learners.

CONCLUSIONS

We have the opportunity to undertake research to close our gap in knowledge about CoS at the PGME level using data emerging from our experiences with CBME. Selecting specific dimensions of CoS will allow research that is necessary to determine that what works at the UME level will also work at the PGME level as we continue to march towards CBME.

Entrustability levels of general internal medicine residents

BACKGROUND

Entrustable professional activities (EPAs) are those activities that a health professional can perform without direct supervision in a defined environment. Bridging the gap between competencies and learning objectives, EPAs have made assessing the performances of health professional more realistic. The main objective of the present study was developing and customizing EPAs for Iranian Internal Medicine Residency Programs.

RESULTS

After reviewing the publications, residency curricula and logbooks, and collecting experts' ideas, the initial list of EPAs was developed. Then, in a focus group, the list was refined, the entrustability level of each residency year was determined, and finally, the EPA-competency cross-tab was established, and in the next step, through a one- round Delphi, the results were validated. Twenty-eight EPAs were developed. Some of them were definitely suitable for the higher levels of residency, such that they had to be accomplished under direct supervision until the end of the program. On the other hand, some of EPAs were those that residents, even from the first year, are expected to perform independently or under indirect supervision. Most of the EPAs cover a wide range of competencies.

CONCLUSION

Determining the entrustability level of each residency year in each EPA as well as the competency- EPA matrix has crucial effect on the quality of the graduates. It seems that our findings are applicable in developing countries like Iran.

EMR Precharting Efficiency in Internal Medicine: A Scoping Review

PURPOSE

The act of precharting, or navigating the EMR to review a patient's recent vitals, labs, notes, and other results, is something that is required of every clinician prior to effective rounding on patients. The purpose of this scoping review is to review the extant literature on precharting.

METHODS

Scholarly data through OVID on Medscape and grey literature were systematically searched with extensive inclusion criteria including the terms "Pre-round" "precharting" as well as "student" "education" or "teach" adjacent to "EMR" or "electronic medical record" or "electronic health record" or "documentation." We collated this with "education, medical, undergraduate," or "Students/medical."

RESULTS

As of September 23, 2020, 241 scholarly articles were identified. No grey literature were identified. Inclusion criteria included full article access, English language, and covering the precharting topic. Seventeen articles met inclusion criteria and were included in the review. These articles included 1 direct observational study, 1 retrospective study, 2 qualitative studies, 5 EMR workshop trainings, 1 perspective piece, 1 curriculum analysis, and 6 articles based on survey measures. Of these articles, the majority were published recently, with 8 of the 17 published since 2018. Summary of the limited existing literature can be distilled into 3 findings: a need for timely EMR data extraction, the potential optimization of EMR workflow, and the benefit of time intensive EMR trainings.

CONCLUSIONS

This scoping review explored the existing scholarly and grey literature to summarize the review of precharting and education surrounding navigating the EMR for medical students as a means of exploring the topic to determine current practices and identify areas of potential improvement.

Building a Shared Mental Model of Competence Across the Continuum: Trainee Perceptions of Subinternships for Residency Preparation

INTRODUCTION

Toward a vision of competency-based medical education (CBME) spanning the undergraduate to graduate medical education (GME) continuum, University of Minnesota Medical School (UMMS) developed the Subinternship in Critical Care (SICC) offered across specialties and sites. Explicit course objectives and assessments focus on internship preparedness, emphasizing direct observation of handovers (Core Entrustable Professional Activity, "EPA," 8) and cross-cover duties (EPA 10).

METHODS

To evaluate students' perceptions of the SICC's and other clerkships' effectiveness toward internship preparedness, all 2016 and 2017 UMMS graduates in GME training (n = 440) were surveyed regarding skill development and assessment among Core EPAs 1, 4, 6, 8, 9, 10. Analysis included descriptive statistics plus chi-squared and Kappa agreement tests.

RESULTS

Respondents (n = 147, response rate 33%) rated the SICC as a rotation during which they gained most competence among EPAs both more (#4, 57% rated important; #8, 75%; #10, 70%) and less explicit (#6, 53%; #9, 69%) per rotation objectives. Assessments of EPA 8 (80% rated important) and 10 (76%) were frequently perceived as important toward residency preparedness. Agreement between importance of EPA development and assessment was moderate (Kappa = 0.40-0.59, all surveyed EPAs).

CONCLUSIONS

Graduates' perceptions support the SICC's educational utility and assessments. Based on this and other insight from the SICC, the authors propose implications toward collectively envisioning the continuum of physician competency.

What can we expect from medical graduates? Empirical survey on the performance of Core EPAs in the first days of residency

BACKGROUND

Core Entrustable Professional Activities (EPAs) have been defined to specify the performance expectations for entering residents worldwide. The content of these EPAs was elaborated and validated primarily via medical expert consent approaches. The present study aims to collect empirical information on the actual task performance and supervision level of entering residents as a complementary methodological approach to enhance the content validity of a set of institutional EPAs.

METHODS

In the summers of 2017 and 2018, Charité medical graduates (n = 720) received a post-graduation survey by mail. The questionnaire covered the performance of Core EPAs, Core procedures and more advanced EPAs. Graduates were asked how frequently they had performed the respective EPAs since the start of residency and under what level of supervision. We expected the large majority of graduates (> 75%) to have performed the Core EPAs and procedures under at least indirect supervision.

RESULTS

In total, 215 graduates (30%) returned the questionnaire, and 131 (18%) surveys could be included in the data analysis. The majority of participants were female (63%) and worked in hospitals (50%) or in university medical centres (30%) across various medical disciplines. Among the Core EPAs, 10 out of 11 tasks had been performed by more than 75% of graduates since the start of residency, 9 under indirect supervision. Regarding the Core procedures, only 3 out of 13 procedures had been performed by the large majority of graduates under indirect supervision, and 10 procedures had not been carried out by at least one-third of participants. Among the 5 advanced EPAs, none of 5 had been performed by more than 75% of the participants since the start of residency, and 4 had been carried out by 50% under indirect supervision.

CONCLUSIONS

The results of this study largely and complementarily confirm the validity of the defined Core EPAs representing realistic expectations for entry into residence at our institution. The low actual performance rate of Core procedures serves to stimulate an institutional discussion on their adjustment to better match the workplace reality.

Encouraging Entrustment: A Qualitative Study of Resident Behaviors That Promote Entrustment

PURPOSE

To explore resident and attending physician perceptions of resident behaviors and skills that demonstrate trustworthiness and promote entrustment by supervisors.

METHOD

Using grounded theory methodology, the authors conducted 3 focus groups with pediatric residents from the Boston Combined Residency Program and 3 focus groups with attending physicians who were either general pediatric hospitalists or other pediatric subspecialists at Boston Children's Hospital and Boston Medical Center in Boston, Massachusetts, from May to December 2018. Data were collected and analyzed iteratively until theoretical saturation was achieved. Three independent reviewers coded each transcript. Codes were grouped into dominant themes to develop a conceptual model.

RESULTS

Twelve residents and 18 attending physicians participated in the focus groups. Participants described actions that they felt actively demonstrated residents' trustworthiness within previously described domains of trustworthiness. Four modifiers emerged that affect a resident's progression from trustworthiness to entrustment: (1) self-management, (2) relationships, (3) self-advocacy, and (4) patient-centeredness. Findings were synthesized into a conceptual model depicting how trainees can promote their own entrustment by supervisors.

CONCLUSIONS

Trainees must actively demonstrate their trustworthiness to be entrusted. This study proposes that trainees can further gain entrustment through self-management, relationships, self-advocacy, and patient-centeredness. When they understand the actions and behaviors that promote entrustment, trainees may be better able to foster autonomy and progress toward more independent clinical practice. These findings add to existing evidence regarding entrustment and provide a novel, actionable framework for trainees to increase their own entrustment.

Status of Entrustable Professional Activities (EPA) Implementation at Colleges of Osteopathic Medicine in the United States and Future Considerations

Context

Competency-based medical education, developmental milestones for residency training, and the single graduate medical education (GME) accreditation system have emerged over the last decade, necessitating new ways to adequately prepare graduates to meet new standards in proficiency, including the 13 Core Entrustable Professional Activities (EPA) for Entering Residency. The American Association of Colleges of Osteopathic Medicine (AACOM) Entrustable Professional Activities (EPA) steering committee has implemented an information-gathering process to provide suggestions for supporting a variety of EPA-related implementation efforts at colleges of osteopathic medicine (COMs) across the country.

Objective

To review the status of EPA implementation at COMs nationally.

Methods

An explanatory mixed-methods design was used to guide information gathering and synthesis of a 41-question survey and interview feedback; the overarching premise of this design was to use qualitative data to build upon initial quantitative findings. This survey was delivered electronically through a link emailed to liaisons at each main, branch, and satellite campus of the 47 schools within the AACOM institutional database. After survey administration, follow-up structured interviews were conducted according to an 18-question script with a purposive sample of 16 institutions with EPA implementation levels ranging from “moderate implementation with reporting” to “full implementation with reporting.” Post-interview, the interview notes were analyzed and results were aggregated for comparison with the original survey findings.

Results

Of the 47 schools surveyed, 42 responded (89.4%). To maintain uniformity in data coding and analysis, 36 of 47 (76.6%) of COMs with independently submitted survey responses were retained in the review. The majority of those respondents (23 of 36; 64%) indicated that their institution was above “somewhat knowledgeable” toward “expert” regarding knowledge of EPAs, but 23 (64%) also indicated “no confidence” or “somewhat confident” regarding EPA implementation. Postinterview results showed that the majority of schools were equally distributed across the “foundational implementation” (10; 28%), “slight implementation” (11; 31%), and “moderate implementation” (11, 31%) categories, with a few schools indicating “no implementation” (2; 5%) or “progressive implementation” (2; 5%).

Conclusion

The results of this review indicate that most osteopathic medical schools are at the early stages of EPA implementation, with emphasis varying by program year in terms of the specific EPAs addressed. Many schools appear engaged in curricular change efforts that will support the advancement of EPA use within their institutions. Faculty development was identified as a continued critical need for a majority of institutions.

Assessing Trainees and Making Entrustment Decisions: On the Nature and Use of Entrustment-Supervision Scales

Clinical teachers are continuously entrusting trainees with care responsibilities in health care settings. Entrustable professional activities employ entrustment decision making as an approach to assessment in the workplace.Various scales have been created to measure "entrustment," all basically expressing the level or type of supervision a trainee requires for safe and high-quality care. However, some of these scales are only weakly related to the purpose of making decisions about the autonomy trainees will be granted. The authors aim to increase understanding about the nature, purpose, and practice of supervision scales aimed at entrustment.After arguing for entrustment as a component of workplace-based assessment, the distinction between ad hoc entrustment decisions (daily decisions in health care settings) and summative entrustment decisions (with a certifying nature) is clarified. Next, the noncontinuous nature of entrustment-supervision (ES) scales, as opposed to most workplace-based assessment scales, is explained. ES scales have ordinal, rather than interval, properties and focus on discrete decisions. Finally, some scales are retrospective ("how much supervision was provided?"), and others are prospective ("how much supervision will be needed in the near future?"). Although retrospective scales reflect observed behavior, prospective scales truly focus on entrustment and ask for more holistic judgment, as they include a broader evaluation and a risk estimation to enable a decision about increase of autonomy.The analysis concludes with a discussion about entrustment for unsupervised practice and supervision of others, as well as the program, context, and specialty specificity of scales.