Competency-based resident education: The United States perspective

The Neurostimulation Appropriateness Consensus Committee (NACC)®: Recommendations for the Mitigation of Complications of Neurostimulation

INTRODUCTION

The International Neuromodulation Society convened a multispecialty group of physicians based on expertise and international representation to establish evidence-based guidance on the mitigation of neuromodulation complications. This Neurostimulation Appropriateness Consensus Committee (NACC)® project intends to update evidence-based guidance and offer expert opinion that will improve efficacy and safety.

MATERIALS AND METHODS

Authors were chosen on the basis of their clinical expertise, familiarity with the peer-reviewed literature, research productivity, and contributions to the neuromodulation literature. Section leaders supervised literature searches of MEDLINE, BioMed Central, Current Contents Connect, Embase, International Pharmaceutical Abstracts, Web of Science, Google Scholar, and PubMed from 2017 (when NACC last published guidelines) to October 2023. Identified studies were graded using the United States Preventive Services Task Force criteria for evidence and certainty of net benefit. Recommendations are based on the strength of evidence or consensus when evidence was scant.

RESULTS

The NACC examined the published literature and established evidence- and consensus-based recommendations to guide best practices. Additional guidance will occur as new evidence is developed in future iterations of this process.

CONCLUSIONS

The NACC recommends best practices regarding the mitigation of complications associated with neurostimulation to improve safety and efficacy. The evidence- and consensus-based recommendations should be used as a guide to assist decision-making when clinically appropriate.

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Entrustable Professional Activities in Surgery: A Review

Importance

Entrustable professional activities (EPAs) compose a competency-based education (CBE) assessment framework that has been increasingly adopted across medical specialties as a workplace-based assessment tool. EPAs focus on directly observed behaviors to determine the level of entrustment a trainee has for a given activity of that specialty. In this narrative review, we highlight the rationale for EPAs in general surgery, describe current evidence supporting their use, and outline some of the practical considerations for EPAs among residency programs, faculty, and trainees.

Observations

An expanding evidence base for EPAs in general surgery has provided moderate validity evidence for their use as well as practical recommendations for implementation across residency programs. Challenges to EPA use include garnering buy-in from individual faculty and residents to complete EPA microassessments and engage in timely, specific feedback after a case or clinical encounter. When successfully integrated into a program's workflow, EPAs can provide a more accurate picture of residents' competence for a fundamental surgical task or activity compared with other assessment methods.

Conclusions and Relevance

EPAs represent the next significant shift in the evaluation of general surgery residents as part of the overarching progression toward CBE among all US residency programs. While pragmatic challenges to the implementation of EPAs remain, the best practices from EPA and other CBE assessment literature summarized in this review may assist individuals and programs in implementing EPAs. As EPAs become more widely used in general surgery resident training, further analysis of barriers and facilitators to successful and sustainable EPA implementation will be needed to continue to optimize and advance this new assessment framework.

Surgical Education

This paper summarizes the proceedings of the joint European Surgical Association ESA/American Surgical Association symposium on Surgical Education that took place in Bordeaux, France, as part of the celebrations for 30 years of ESA scientific meetings. Three presentations on the use of quantitative metrics to understand technical decisions, coaching during training and beyond, and entrustable professional activities were presented by American Surgical Association members and discussed by ESA members in a symposium attended by members of both associations.

Enhancing the Value of Surgical Entrustable Professional Activities through Integrative Learning Analytics

OBJECTIVE

To demonstrate the value of integrating surgical resident Entrustable Professional Activity (EPA) data into a learning analytics platform that provides meaningful feedback for formative and summative decision-making.

DESIGN

Description of the Surgical Entrustable Professional Activities (SEPA) analytics dashboard, and examples of summary analytics and intuitive display features.

SETTING

Department of Surgery, University of Wisconsin Hospital and Clinics.

PARTICIPANTS

Surgery residents, faculty, and residency program administrators.

RESULTS

We outline the major functionalities of the SEPA dashboard and offer concrete examples of how these features are utilized by various stakeholders to support progressive entrustment decisions for surgical residents.

CONCLUSIONS

Our intuitive analytics platform allows for seamless integration of SEPA microassessment data to support Clinical Competency Committee (CCC) decisions for resident evaluation and provides point of training feedback to faculty and trainees in support of progressive autonomy.

Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation

Objectives

Workplace-based assessment (WBA) is a key requirement of competency-based medical education in postgraduate surgical education. Although simulated workplace-based assessment (SWBA) has been proposed to complement WBA, it is insufficiently adopted in surgical education. In particular, approaches to criterion-referenced and automated assessment of intraoperative surgical competency in contextualized SWBA settings are missing.Main objectives were (1) application of the universal framework of intraoperative performance and exemplary adaptation to spine surgery (vertebroplasty); (2) development of computer-assisted assessment based on criterion-referenced metrics; and (3) implementation in contextualized, team-based operating room (OR) simulation, and evaluation of validity.

Design

Multistage development and assessment study: (1) expert-based definition of performance indicators based on framework's performance domains; (2) development of respective assessment metrics based on preoperative planning and intraoperative performance data; (3) implementation in mixed-reality OR simulation and assessment of surgeons operating in a confederate team. Statistical analyses included internal consistency and interdomain associations, correlations with experience, and technical and non-technical performances.

Setting

Surgical simulation center. Full surgical team set-up within mixed-reality OR simulation.

Participants

Eleven surgeons were recruited from two teaching hospitals. Eligibility criteria included surgical specialists in orthopedic, trauma, or neurosurgery with prior VP or kyphoplasty experience.

Main outcome measures

Computer-assisted assessment of surgeons' intraoperative performance.

Results

Performance scores were associated with surgeons' experience, observational assessment (Objective Structured Assessment of Technical Skill) scores and overall pass/fail ratings. Results provide strong evidence for validity of our computer-assisted SWBA approach. Diverse indicators of surgeons' technical and non-technical performances could be quantified and captured.

Conclusions

This study is the first to investigate computer-assisted assessment based on a competency framework in authentic, contextualized team-based OR simulation. Our approach discriminates surgical competency across the domains of intraoperative performance. It advances previous automated assessment based on the use of current surgical simulators in decontextualized settings. Our findings inform future use of computer-assisted multidomain competency assessments of surgeons using SWBA approaches.

Implementation of entrustable professional activities into fellowship council accredited programs: a pilot project

OBJECTIVE

The Fellowship Council (FC) is transitioning to a competency-based medical education (CBME) model, including the introduction of Entrustable Professional Activities (EPAs) for training and assessment of Fellows. This study describes the implementation process employed by the FC during a ten-month pilot project and presents data regarding feasibility and perceived value.

METHODS

The FC coordinated the development of EPAs in collaboration with the sponsoring societies for Advanced GI/MIS, Bariatrics, Foregut, Endoscopy and Hepatopancreaticobiliary (HPB) fellowships encompassing the preoperative, intraoperative, and postoperative phases of care for key competencies. Fifteen accredited fellowship programs participated in this project. The assessments were collected through a unique platform on the FC website. Programs were asked to convene a Clinical Competency Committee (CCC) on a quarterly basis. The pilot group met monthly to support and improve the process. An exit survey evaluated the perceived value of EPAs.

RESULTS

The 15 participating programs included 18 fellows and 106 faculty. A total of 655 assessments were initiated with 429 (65%) completed. The average (SD) number of EPAs completed for each fellow was 24(18); range 0-72. Intraoperative EPAs were preferentially completed (71%). The average(SD) time for both the fellow and faculty to complete an EPA was 27(78) hours. Engagement increased from 39% of fellows completing at least one EPA in September to 72% in December and declining to 50% in May. Entrustment level increased from 6% of EPAs evaluated as "Practice Ready" in September to 75% in June. The exit survey was returned by 63% of faculty and 72% of fellows. Overall, 46% of fellows and 74% of program directors recommended full-scale implementation of the EPA framework.

CONCLUSION

A competency-based assessment framework was developed by the FC and piloted in several programs. Participation was variable and required ongoing strategies to address barriers. The pilot project has prepared the FC to introduce CBME across all FC training programs.

Transition to practice curriculum for general internal medicine physicians: scoping review and Canadian national survey

BACKGROUND

There remains a paucity of evidence for curricula for the transition to practice (TTP) stage of Competence by Design internal medicine (IM) training programs. Current entrustable professional activities are based on expert consensus rather than robust subspecialty-specific needs assessment.

METHODS

A scoping review was completed to identify studies with TTP focus. A national survey was conducted to identify transition experiences for general internal medicine physicians. Results were assessed by grounded theory analysis to identify core topics for TTP curricula.

RESULTS

Neither scoping review nor national survey identified TTP topics related to the CanMEDS Medical Expert role. Scoping Review: 41 relevant studies were identified. Most (97.6%) were from North America. The most common study types were observational (survey) or curriculum (13/41 31.7% for each). Only two studies were exclusively in IM, and the most common subspecialty studied was surgical (13/41, 31.7%). The most common TTP topics were mentorship, billing and coding, practice management, negotiating contract and job, and financial aspects of practice. National Survey: There were 44 respondents, with the majority (25/44, 56.8%) having completed an IM subspecialty fellowship. Most (38/44) completed medical school in Canada, and most were from academic practice settings (33/44, 75.0%). The most common TTP topics were billing and coding, personal financial planning, practice management, work-life balance and mentorship. Grounded Theory Analysis: There were six themes that encompassed all TTP topics from the scoping review and national survey, being (i) building a career, (ii) continuing professional development, (iii) expectations of the profession, (iv) practice management, (v) Life, health and well-being and (vi) clinical skills. Curriculum competencies and resources for curriculum development were provided.

CONCLUSIONS

This study identifies topics critical for curricula development for IM transition to practice. Further research is required to evaluate effectiveness of curricula including topics and themes developed from this scoping review and national survey.

Resident self-assessment of common endocrine procedures

BACKGROUND

General surgery residency graduates are expected to be proficient in straightforward endocrine operations. This study aimed to elucidate residents' self-assessment of their ability to perform common endocrine procedures.

METHODS

A fourteen-question survey was emailed to general surgery residents from seven U.S. residency programs regarding their self-assessed ability to perform each step of a straightforward thyroidectomy and parathyroidectomy. Demographics and perceived ability to perform the various procedures were collected.

RESULTS

A minority of respondents (17, 27.9%) agreed they could complete a straightforward thyroidectomy for benign disease, with only 11.7% (n = 7) agreeing they could complete a straightforward thyroidectomy for malignant disease. 26.2% (n = 16) of respondents agreed they could complete a straightforward parathyroidectomy. Completed number of cases was significantly associated with greater self-assessed ability to perform the endocrine operations (p = 0.02).

CONCLUSIONS

Most general surgery residents surveyed did not feel capable of performing common, straightforward endocrine procedures. Although confidence in operative ability increased with PGY-level and number of cases completed, the majority of PGY-5 residents still did not feel able to perform a thyroidectomy for malignant disease unassisted.

A 20-year review of surgical training case logs: Is general surgery still general?

BACKGROUND

Surgical training has undergone many facets of restructuring over the most recent decades, with critiques of the quality and variability of training as well as the competency of recent graduates. This study examines the changes in surgical training in operative volume and breadth in the past 2 decades.

METHODS

The Accreditation Council for Graduate Medical Education Case Log Statistics Reports from 1999 to 2019 were reviewed. Case logs were grouped into defined case categories and group levels of postgraduate training. Descriptive analyses and multiple linear regressions were performed.

RESULTS

General surgery residents are graduating with 10.7% more cases, owing to increases in mostly junior year cases (P < .001). The breadth of specialty cases has decreased, while there was an increase in alimentary and abdominal cases to 58.4% from 47.2% 20 years ago. A decrease in vascular surgery cases from 19.9% to 10.7% of all cases was noted. Analysis of the distribution of defined categories showed right skewness in many categories with mode being much lower than reported mean.

CONCLUSION

Evaluation of trends, despite residents graduating with higher case volume than the minimum required, shows that the breadth and variety of cases has narrowed significantly in the past 20 years, providing a case for general surgery training restructuring.

Aligning Urology Residency Training With Real-World Workforce Needs

OBJECTIVE

Research suggests recently graduated urology residents do not feel ready for independent practice. We conducted a study to determine if Accreditation Council for Graduate Medical Education (ACGME) minimum case requirements, resident case logs, and graduating resident perceived readiness for practice are aligned with the procedural demand and needs of the current urology workforce.

DESIGN

Correlative study comparing the association between (1) workforce demand and ACGME case requirements, and (2) workforce demand and perceived resident competency. Three distinct datasets were used; (1) the 2017 Medicare Part B National Summary Data File; (2) the 2017 National Data Report published by the ACGME; and (3) a graduating resident survey from Okhunov et al. SETTING: N/A.

PARTICIPANTS

N/A.

RESULTS

In 2017, there were a total of 6,784,696 urologic cases performed through Medicare. We found nonsignificant positive associations between resident case logs (rho = 0.16, p = 0.5784), ACGME minimum procedure requirements (rho = 0.42, p = 0.1255), and Medicare procedural demand. Our 15 index procedures accounted for 21.1% (n = 1,431,775) of all Medicare cases, with a median number of 7706 procedures. Endopyelotomy was the least common procedure (n = 98), while cystoscopy was the most common (n = 980,623). Medicare case volume was positively correlated with graduating residents' procedural confidence (r = 0.86, p < 0.0001). We identified four categories with varied alignment of training and demand: (1) high volume and high confidence, (2) high volume and low confidence, (3) low volume and high confidence, and (4) low volume and low confidence.

CONCLUSIONS

Optimizing urology residency training is time-sensitive and important. Using national Medicare data coupled with recently graduated urology resident survey results, we provide a guiding framework for improving the alignment of training with workforce demand. Informed by these results, we recommend altering training requirements to reflect these needs.

APDS Consensus Statement: Ideal Senior Medical Student Experiences for Preparedness for General Surgery Internship

Guide optimal standards on ideal senior medical student experiences for preparedness for general surgery internship DESIGN: Work product of task force, approved by the Association of Program Directors in Surgery CONCLUSION: General surgery rotations should mirror the learning and working environment of a surgical intern. Opportunities should mimic the next phase of learning to help guide informed decisions regarding entrustability for entry into residency training. These opportunities will also help identify students who may have an aptitude for pursuing a general surgery internship. Students should achieve entrustability in Association of American Medical Colleges Core Entrustable Professional Activities (EPAs); curricula should align Core EPAs and modified American Board of Surgery EPAs to guide essential general surgery components. Experiences should include required night, holiday, and/or weekend shifts, a dedicated critical care experience, and a resident preparatory curriculum focusing on nontechnical and essential technical skills. We encourage the opportunity for additional surgical mentorship and subspecialty experience through Surgical Interest Groups or Surgical Honors or Specialty Tracks.

Simulation in surgical education

Simulation is becoming an important tool in surgical education. Surgical faculty have been forced to modify how they teach technical skills. Instead of a complete reliance on teaching in the operating room, a structured curriculum and dedicated time in the simulation center are being used in many centers. Some of the advantages of this approach include the ability to learn and practice new procedures in a safe and nurturing environment. The disadvantages include the significant cost of virtual reality simulators and the competition, between various training programs, to gain access to simulation.

A longitudinal cadaver-based simulation curriculum creates sustainable increases in resident confidence and exposure to fundamental techniques: Results of a 5-year program evaluation

INTRODUCTION

For the past five years, our surgical residency program has led a cadaver-based simulation course focused on fundamental surgical maneuvers. This study aimed to quantify the impact of this course on resident exposure to surgical skills and longitudinal impact on resident education.

METHODS

General surgery residents participated in an annual cadaver-based simulation curriculum. Participants completed surveys regarding improvements in knowledge and confidence; these results were stratified between course iterations (P1: 2 years, 2014-15; P2: 3 years, 2016-2018).

RESULTS

Residents reported a sustained increase in knowledge of anatomy and technical dissection, confidence in performing operative skills independently, and exposure to operative skills that were otherwise not encountered in clinical rotations. Junior residents demonstrated an increase in gaining skills they would otherwise not achieve (87% vs. 98%, p = 0.028) and confidence to safely perform these procedures in the clinical setting (94% vs. 100%, p = 0.077).

CONCLUSION

This annual, longitudinal cadaver-based skills course focused on fundamental maneuvers demonstrates a sustained impact in resident and faculty surgical confidence in resident's operative skills as a component of a longitudinal simulation curriculum to enhance competency-based promotion.

Competency Assessment for Mediastinal Mass Resection and Thymectomy: Design and Delphi Review Process

OBJECTIVES

Mediastinal mass resection and thymectomy are complex and related operations that are core components of competency for a general thoracic surgeon and an important learning objective for thoracic surgery trainees. This study aimed to design a combined competency assessment instrument for mediastinal mass resection and thymectomy.

DESIGN

A comprehensive competency assessment instrument was designed by a process of logical analysis by 3 expert thoracic surgeons with an interest in mediastinal surgery. The instrument was then assessed and refined using a modified Delphi process.

SETTING

The Delphi questionnaire was distributed to all members of the Canadian Association of Thoracic Surgeons in 2018 to 2019.

PARTICIPANTS

The first round of the Delphi review was completed by 58 respondents (response rate 43.9%). Respondents represented all Canadian provinces with a wide range of clinical experience and a high rate of involvement in resident education.

RESULTS

A first draft of the competency assessment instrument included 42 steps in 6 categories. A total of 3 rounds of Delphi review were performed. Cronbach's alpha for the final round was 0.83. Ultimately, 29 items were retained from the original instrument and two modified and three new items were added. The final instrument has 34 steps in 5 categories.

CONCLUSIONS

A nationwide consensus was established on the key components of assessing competence to perform mediastinal mass resection and thymectomy. The resulting instrument could be used to guide competency based assessments of thoracic surgeons and trainees.