The surgical consult entrustable professional activity (EPA): Defining competence as a basis for evaluation

Decision-Making Confidence of Clinical Competency Committees for Entrustable Professional Activities

Importance

A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs' clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known.

Objective

To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions.

Design, Setting, and Participants

This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023.

Exposure

Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle.

Main Outcomes and Measures

The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence.

Results

Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association.

Conclusions and Relevance

In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.

Evaluating competency-based medical education: a systematized review of current practices

BACKGROUND

Few published articles provide a comprehensive overview of the available evidence on the topic of evaluating competency-based medical education (CBME) curricula. The purpose of this review is therefore to synthesize the available evidence on the evaluation practices for competency-based curricula employed in schools and programs for undergraduate and postgraduate health professionals.

METHOD

This systematized review was conducted following the systematic reviews approach with minor modifications to synthesize the findings of published studies that examined the evaluation of CBME undergraduate and postgraduate programs for health professionals.

RESULTS

Thirty-eight articles met the inclusion criteria and reported evaluation practices in CBME curricula from various countries and regions worldwide, such as Canada, China, Turkey, and West Africa. 57% of the evaluated programs were at the postgraduate level, and 71% were in the field of medicine. The results revealed variation in reporting evaluation practices, with numerous studies failing to clarify evaluations' objectives, approaches, tools, and standards as well as how evaluations were reported and communicated. It was noted that questionnaires were the primary tool employed for evaluating programs, often combined with interviews or focus groups. Furthermore, the utilized evaluation standards considered the well-known competencies framework, specialized association guidelines, and accreditation criteria.

CONCLUSION

This review calls attention to the importance of ensuring that reports of evaluation experiences include certain essential elements of evaluation to better inform theory and practice.

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Mortality and Severe Complications Among Newly Graduated Surgeons in the United States

OBJECTIVE

To evaluate severe complications and mortality over years of independent practice among general surgeons.

BACKGROUND

Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons.

METHODS

We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery. Generalized additive mixed models were used to assess the association between surgeon years in practice and 30-day outcomes while adjusting for differences in patient, hospital, and surgeon characteristics.

RESULTS

The cohort included 1,329,358 operations performed by 14,399 surgeons. In generalized mixed models, the relative risk (RR) of mortality was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [5.5% (95% CI: 4.1%-7.3%) vs 4.7% (95% CI: 3.5%-6.3%), RR: 1.17 (95% CI: 1.11-1.22)]. Similarly, the RR of severe complications was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [7.5% (95% CI: 6.6%-8.5%) versus 6.9% (95% CI: 6.1%-7.9%), RR: 1.08 (95% CI: 1.03-1.14)]. When stratified by individual operation, 21 operations had a significantly higher RR of mortality and all 26 operations had a significantly higher RR of severe complications in the first compared with the 15th year of practice.

CONCLUSIONS

Among general surgeons performing common operations, rates of mortality and severe complications were higher among newly graduated surgeons compared with later career surgeons.

Advancing Urology Resident Surgical Autonomy

PURPOSE OF REVIEW

This paper aims to survey current literature on urologic graduate medical education focusing on surgical autonomy.

RECENT FINDINGS

Affording appropriate levels of surgical autonomy has a key role in the education of urologic trainees and perceived preparedness for independent practice. Recent studies in surgical resident autonomy have demonstrated a reduction in autonomy for trainees in recent years. Efforts to advance the state of modern surgical training include creation of targeted curricula, enhanced with use of surgical simulation, and structured feedback. Decline in surgical autonomy for urology residents may influence confidence after completion of their residency. Further study is needed into the declining levels of urology resident autonomy, how it affects urologists entering independent practice, and what interventions can advance autonomy in modern urologic training.

Differences in perceptions of capability, autonomy, and expectations between residents and surgical team members in executing EPAs in Ethiopian medical education

Background

Entrustable Professional Activities (EPAs) are units of professional practice that are defined as tasks or responsibilities that are entrusted to an unsupervised execution by a trainee. In 2021, a framework of 29 EPAs was developed for surgical residency training programs in Ethiopia, with the goal of residents being able to perform independently by the time they graduate. However, studies show that surgical residents lack confidence and are unable to execute EPAs autonomously upon graduation, and concerns have been raised about graduate competencies in EPA execution. The goal of this research is to assess how surgical team members judge/perceive residents' performance in executing these EPAs autonomously at the time of graduation and how residents rate their own capability and autonomy in executing EPAs in order to systematically introduce and implement EPAs in Ethiopian medical education.

Methods

A survey was conducted in the Departments of Surgery at four residency training institutions in Ethiopia. All eligible surgical team members and final-year general surgery residents were invited to participate. Surgical team members were asked to rate the observed performance of a group of graduating surgical residents in each of the 29 EPAs, and residents were asked to rate their own capability in executing EPAs. The analysis focused on variations in performance ratings between surgical team members and residents, as well as across surgical team members.

Results

A total of 125 surgical team members and 49 residents participated in this study. Residents rate their competence in performing these EPAs higher than surgical team members, mean 4.2 (SD = 0.63) vs. 3.7 (SD = 0.9). A statistically significant difference in perceptions of capability, autonomy, and expectations in executing EPAs was observed between the two groups of study (p = 0.03, CI: 0.51-0.95), as well as within surgical team members (p < 0.001).

Conclusions

Differences in perceptions of capability, autonomy, and expectations between residents and surgical team members, as well as within faculty members, were seen in executing EPAs. There were concerns about graduate surgical residents' competence to execute EPAs autonomously at the time of graduation. Surgical team members perceived that a set of graduating surgical residents are not yet safe to perform these EPAs independently (without supervision) and still requires distant supervision.

An analysis of vascular surgery education publications reveals an educational shortage

BACKGROUND

As vascular surgery training continues to evolve with the growth of integrated vascular surgery residency (0+5) programs and the consolidation of fellowship programs, optimizing all aspects of the education for vascular surgery trainees, both fellows and 0+5 residents, has become increasingly important. In the present study, we aimed to determine the prevalence, quality, and content of vascular surgery education publications across journals.

METHODS

Journal websites (n = 26) and PubMed were queried to identify vascular surgery education publications from 2012 to 2021. The publications were organized into 11 content categories: (1) curriculum, (2) simulation, (3) trainee assessment, (4) program evaluation, (5) wellness/burnout, (6) diversity/inclusion, (7) mentorship/career, (8) case outcomes, (9) perceptions of training, (10) social media, and (11) other. Publication interactivity and quality were measured via PlumX data and Medical Education Research Study Quality Instrument scores. The data were analyzed via univariate analysis and linear regression.

RESULTS

A total of 115 vascular surgery education publications (0.2% [interquartile range (IQR), 0.04%-0.5%] of total publications) were identified from the selected journals. The Journal of Vascular Surgery had the highest proportion (0.8%) of vascular surgery education publications, followed by the Journal of Surgical Education (0.7%) and Annals of Vascular Surgery (0.6%). Vascular surgery journals constituted most (79%) of the publications. Of the authors, 15% (IQR, 0%-25%) had a master's or doctorate degree in education. Senior authors were more often identified as male gender (77%), and more first authors (41%) were identified as female gender. An interactivity analysis showed that there were 10.3 citations (IQR, 12), 33.1 captures (IQR, 34), and 8.4 social media interactions (IQR 14) per publication. The educational quality had a median Medical Education Research Study Quality Instrument score of 11 (IQR, 9-12.5), with 49% of publications having a score greater than the median. Publications on training (44% curriculum and 20% simulation) were significantly more frequent than other topics (P < .001), with no change in the publication content over 10 years (P = .29). The volume of vascular surgery education publications did not change during the study period (P = .13) despite the ongoing changes in the educational environment.

CONCLUSIONS

Despite the increasing importance placed on vascular surgery education by national vascular societies, publications on vascular surgery education have remained sparse among all journals. Also, the vascular surgery educational content has not changed during the past 10 years, with a primary focus on curriculum and simulation training. Further promotion of vascular surgery educational research is required to increase the quality, volume, and diversity of education publications.

Competency-Based Education: Will This be the New Training Paradigm in Plastic Surgery?

The Accreditation Council for Graduate Medical Education created the "Next Accreditation System" in 2013 requiring residents to meet educational milestones based on core competencies over the course of their training. The 6 core competencies include patient care and technical skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. Since the traditional time-based model requires a predetermined length of training irrespective of learning style, pace, or activity, a competency-based model is appealing because it refocuses education on deliberate and relevant skills acquisition and retention. Plastic surgery has been slowly transitioning to competency-based education (CBE), thereby permitting residents to learn at their own pace to master each competency. We performed a nonsystematic literature review of the efficacy of CBE and implementation efforts, particularly within plastic surgery. The literature revealed perceived barriers to implementation, as well as the nuts and bolts of implementation. We highlighted possible solutions and training tools with practical applications in plastic surgery. Success of CBE in plastic surgery requires instituting a transparent process that involves continuously piloting multiple assessment tools and a discussion of related costs. CBE may be particularly appealing for trainees focused on further training in craniofacial or pediatric plastic surgery after completion of an integrated or independent training program in plastic surgery to allow them to focus on their career interests once competence is achieved in the core skills required of a plastic surgeon.

Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents

BACKGROUND

A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency.

STUDY DESIGN

This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings.

RESULTS

From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05).

CONCLUSIONS

The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.

Development of an Entrustable Professional Activities (EPA) Framework to Inform Surgical Residency Training Programs in Ethiopia: A Three-round National Delphi Method Study

BACKGROUND

Entrustable Professional Activities (EPAs) have been proposed as a means to translate competencies into clinical practice. Although EPAs for residency training have become available, 1 set of core EPAs cannot automatically be transferred from one context to another due to cultural variability. Further, there is a lack of African- and Asian-based EPA development and implementation studies. We developed an end-of-training EPAs framework to inform surgical residency training programs in the local context of Ethiopian medical education.

METHODS

A three-round Delphi method was used to establish consensus about important surgical EPAs among experts. A total of 136 experts representing all surgical residency training institutions in Ethiopia were invited to participate. Round 1 & 2 consisted of senior expert panelists (n = 8) to identify potential EPAs and determine the content validity. Round 3 consisted of a survey (n = 128) to further validate the identified EPAs by attending surgeons who work with them. Each EPA had to achieve at least 80% or higher agreement among experts to be considered having acceptable content validity.

RESULTS

In round 1, a total of 272 EPAs were proposed, reduced, and grouped to 39 consented EPAs. In round 2, the same experts rated each EPA's relevance, resulting in 32 EPAs with a satisfactory item-level content validity index (I-CVI > 0.83). Overall, in the survey in round 3, 29 EPAs met the standard criterion for acceptability (S-CVI/Ave = 0.90) and achieved a high degree of final consensus (ICC = 0.998, 95% CI [0.996, 0.999]; (F = 439.2, p < 0.0001).

CONCLUSIONS

The framework of 29 validated and accepted EPAs can guide future surgical residency training programs in the Ethiopian medical education context. The framework allows programs to move from a time-dependent to an outcome-based model and transforms traditional assessment into entrustment decisions. Thus, the use of the framework can improve the quality of training and patient care in Ehtiopia.

Attendings' Perceptions of Authentic Evaluation Criteria for Effective Surgical Consults

OBJECTIVE

The authors aimed to investigate faculty evaluation criteria for an effective oral surgical presentation in actual patient care contexts.

DESIGN

We conducted a 2-step observation-based qualitative study. Residents audiotaped oral presentations of a surgical consult to an attending. Evaluation panels listened to the recordings and discussed to develop joint feedback for the resident. The panel discussions were recorded and served as the data source for this study. We analyzed the data following the grounded theory approach using open coding and axial coding.

SETTING

The study setting was at Southern Illinois University School of Medicine, a 5-year general surgery residency program in Springfield, Illinois.

PARTICIPANTS

Thirteen residents out of 19 in the program participated by virtue of having submitted recordings of a patient care consult presentation via phone. Evaluation panels consisted of general surgery academic and community faculty, as well as senior residents.

RESULTS

Several criteria for effective oral presentations emerged that have rarely been discussed in prior literature. Themes included: (1) The strategic opening is critical as it "sets the stage" and frames how the attending will listen. Situational factors, such as consideration of time of the day and urgency, should be accounted for in the opening. (2) A deductive structure defines the relevance of the presented information. Clinical judgement should precede supporting evidence. Attending physicians perceive important information as unnecessary if provided outside of this framework. (3) Established trust between a resident and a surgeon determines the level of detail expected of the presenting resident. With increasing trust, surgeons expect residents to present fewer details; if too much detail is included, the presentation may be assessed as ineffective. (4) Surgical descriptions are appreciated for their value in promoting the attending's visualization or mental picture of the patient condition. (5) Oral emphasis using voice tone and pace can be helpful for capturing attending attention.

CONCLUSIONS

These findings can be utilized to improve the current training program and assessment rubrics toward contextualized work-based assessment practices in surgery. Oral patient presentation skills are neither static nor universal, but fluid and reflexive, based on trust, and situational factors.

Fundamentos para un currículo nacional en Cirugía General basado en competencia: Consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía

Frente a los desafíos de la Educación Basada en Competencia, este consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía informa sobre las mínimas competencias profesionales esperadas del cirujano egresado de los veinte programas de Especialización en Cirugía General en Colombia. Un total de 105 profesores de los programas de especialización evaluaron tres áreas de competencia profesional: 1) atributos profesionales generales del residente durante su formación, 2) competencias prácticas (procedimientos quirúrgicos) que los residentes deben realizar al final de su entrenamiento y 3) Actividades Profesionales Confiables (APC) que los residentes deben ejecutar sin supervisión al final de su entrenamiento. Los resultados informan un alto nivel de consenso en el 100 % los atributos profesionales generales y APC, y del 75 % en diferentes procedimientos quirúrgicos. El consenso abre la puerta para el desarrollo de un currículo nacional de la especialidad y tiene implicaciones para la práctica educativa e investigación futura.

Implementation of Entrustable Professional Activities into a General Surgery Residency

OBJECTIVE

Concerns over resident ability to practice effectively after graduation have led to the competency-based medical education movement. Entrustable professional activities (EPAs) may facilitate competency-based medical education in surgery, but implementation is challenging. This manuscript describes 1 strategy used to implement EPAs into an academic general surgery residency.

DESIGN, SETTING, PARTICIPANTS

A mobile application was developed incorporating 5 EPAs developed by the American Board of Surgery; residents and faculty from the Departments of Surgery, Emergency Medicine, and Hospital Medicine at a single tertiary care center were trained in its use. Entrustment levels and free text feedback were collected. Self-assessment was paired with supervisor assessment, and faculty assessments were used to inform clinical competency committee entrustment decisions. Feedback was regularly solicited from app users and results distributed on a monthly basis.

RESULTS

One thousand seven hundred and twenty microassessments were collected over the first 16 months of implementation; 898 (47.8%) were performed by faculty with 569 (66.0%) matched pairs. Engagement was skewed with small numbers of high performers in both resident and faculty groups. Continued development of resident and faculty was required to sustain engagement with the program. Nonsurgical specialties contributed significantly to resident assessments (496, 28.8%).

CONCLUSIONS

EPAs are being successfully integrated into the assessment framework at our institution. EPA implementation in surgery residency is a long-term process that requires investment, but may address limitations in the current assessment framework.