Targeted surgical coaching can improve operative self-assessment ability: A single-blinded nonrandomized trial

Video-Based Coaching for Laparoscopic Salpingectomy: A Randomized Controlled Trial

OBJECTIVE

To evaluate the effect of video-based coaching on the gynecology resident performance of laparoscopic salpingectomy.

METHODS

PGY-1 and PGY-2 residents were randomized before their gynecology rotations to standard gynecology curriculum (control group) or standard curriculum plus two video-coaching sessions by a fellowship-trained minimally invasive gynecologic surgeon (VBC group). Residents were video recorded intraoperatively performing three unilateral laparoscopic salpingectomies. Participants in the VBC group were coached between the procedures. The primary outcome was the improvement in modified GOALS (Global Operative Assessment of Laparoscopic Skills) and OSA-LS (Objective Structured Assessment of Laparoscopic Salpingectomy) scores, compared with baseline, in the VBC and control groups, with videos independently graded by three blinded minimally invasive gynecologic surgeons. A minimum sample size of 18 participants (nine per group) was needed to achieve 90% power to detect a difference of 5.0 points.

RESULTS

From October 2021 to December 2022, 28 PGY-1 and PGY-2 residents completed the study with 14 participants per group. Baseline characteristics were similar between groups. In the VBC group, modified GOALS scores significantly improved by 3.0 points from video 1 to video 2 ( P =.04) and by 3.2 points from video 1 to video 3 ( P =.02). Modified OSA-LS scores also increased significantly in the VBC group, by 6.1 points from video 1 to video 3 ( P =.02). In the control group, modified GOALS and OSA-LS scores improved from baseline but were not significant ( P =.2, P =.5). Video-based coaching also enhanced resident comfort and confidence in performing laparoscopic surgery.

CONCLUSION

Video-based coaching improves resident performance of laparoscopic salpingectomy and can be used as an adjunct to resident surgical training.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov , NCT05103449.

Implementation and Validation of an Automated, Longitudinal Robotic Surgical Evaluation and Feedback Program at a High-volume Center and Impact on Training

Background

Surgical education lacks a standardized, proficiency-based approach to evaluation and feedback.

Objective

To assess the implementation and reception (ie, feasibility) of an automated, standardized, longitudinal surgical skill assessment and feedback system, and identify baseline trainee (resident and fellow) characteristics associated with achieving proficiency in robotic surgery while learning robotic-assisted laparoscopic prostatectomy.

Design setting and participants

A quality improvement study assessing a pilot of a surgical experience tracking program was conducted over 1 yr. Participants were six fellows, eight residents, and nine attending surgeons at a tertiary cancer center.

Intervention

Trainees underwent baseline self-assessment. After each surgery, an evaluation was completed independently by the trainee and attending surgeons. Performance was rated on a five-point anchored Likert scale (trainees were considered "proficient" when attending surgeons' rating was ≥4). Technical skills were assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and Prostatectomy Assessment and Competency Evaluation (PACE).

Outcome measurements and statistical analysis

Program success and utility were assessed by evaluating completion rates, evaluation completion times, and concordance rates between attending and trainee surgeons, and exit surveys. Baseline characteristics were assessed to determine associations with achieving proficiency.

Results and limitations

Completion rates for trainees and attending surgeons were 72% and 77%, respectively. Fellows performed more steps/cases than residents (median [interquartile range]: 5 [3-7] and 3 [2-4], respectively; p < 0.01). Prior completion of robotics or laparoscopic skill courses and surgical experience measures were associated with achieving proficiency in multiple surgical steps and GEARS domains. Interclass correlation coefficients on individual components were 0.27-0.47 on GEARS domains.

Conclusions

An automated surgical experience tracker with structured, longitudinal evaluation and feedback can be implemented with good participation and minimal participant time commitment, and can guide curricular development in a proficiency-based education program by identifying modifiable factors associated with proficiency, individualizing education, and identifying improvement areas within the education program.

Patient summary

An automated, standardized, longitudinal surgical skill assessment and feedback system can be implemented successfully in surgical education settings and used to inform education plans and predict trainee proficiency.

Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial

BACKGROUND

The value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees.

METHODS

This parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment.

RESULTS

Of 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool.

CONCLUSION

This pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.

Standardization and Implementation of a Surgical Coaching Model for Pediatric Urology

To bridge gaps in proficiency and encourage life-long learning following training, coaching models have been utilized in multiple surgical fields; however, not within pediatric urology. In this review of our methodology, we describe the development of a coaching model at a single institution. In our initial experience, the perceived most beneficial aspect of the program was the goal setting process with logistics around debriefs being the most challenging. With our proposed coaching study, we aim to develop a model based upon prior coaching frameworks,1,2 that is feasible and universally adaptable to allow for further advancement of surgical coaching, particularly within the field of pediatric urology.

Association of Gender and Operative Feedback Quality in Surgical Residents

OBJECTIVE

Feedback is critical for learning, however, gender differences exist in the quality of feedback that trainees receive. For example, narrative feedback on surgical trainees' end-of-block rotations differs based on trainee-faculty gender dyads, with female faculty giving higher quality feedback and male trainees receiving higher quality feedback. Though this represents evidence of gender bias in global evaluations, there is limited understanding of how much bias might be present in operative workplace-based assessments (WBAs). In this study, we explore the quality of narrative feedback among trainee-faculty gender dyads in an operative WBA.

DESIGN

A previously validated natural language processing model was used to examine instances of narrative feedback and assign a probability of being characterized as high quality feedback (defined as feedback which was relevant as well as corrective and/or specific). A linear mixed model was performed, with probability of high quality feedback as the outcome, and resident gender, faculty gender, PGY, case complexity, autonomy rating, and operative performance rating as explanatory variables.

PARTICIPANTS

Analyses included 67,434 SIMPL operative performance evaluations (2,319 general surgery residents, 70 institutions) collected from September 2015 through September 2021.

RESULTS

Of 36.3% evaluations included narrative feedback. Male faculty were more likely to provide narrative feedback compared to female faculty. Mean probabilities of receiving high quality feedback ranged from 81.6 (female faculty-male resident) to 84.7 (male faculty-female resident). Model-based results demonstrated that female residents were more likely to receive high quality feedback (p < 0.01), however, there was no significant difference in probability of high quality narrative feedback based on faculty-resident gender dyad (p = 0.77).

CONCLUSIONS

Our study revealed resident gender differences in the probability of receiving high-quality narrative feedback following a general surgery operation. However, we found no significant differences based on faculty-resident gender dyad. Male faculty were more likely to provide narrative feedback compared to their female colleagues. Further research using general surgery resident-specific feedback quality models may be warranted.

Surgical Education

This paper summarizes the proceedings of the joint European Surgical Association ESA/American Surgical Association symposium on Surgical Education that took place in Bordeaux, France, as part of the celebrations for 30 years of ESA scientific meetings. Three presentations on the use of quantitative metrics to understand technical decisions, coaching during training and beyond, and entrustable professional activities were presented by American Surgical Association members and discussed by ESA members in a symposium attended by members of both associations.

Introduction to surgical coaching

Surgical coaching is a new approach to continuous professional development (CPD). Advocates claim that coaching improves surgical and non-technical skills, leading to enhanced performance and reduced stress and burnout. Widespread acceptance of coaching in the surgical community is limited due to a paucity of evidence, surgery-specific obstacles and lack of resources. This narrative review introduces what 'surgical coaching' is and describes its origins. Coaching techniques are based on live or video observation of procedures, followed by collaborative analysis, reflection and goal planning in a non-competitive setting aided by an expert, peer or lay coach. Technical skills improvements have been reported using coaching techniques, yet long-term influence on non-technical competencies, overall performance and surgeon wellbeing are largely hypothetical. There are clear potential benefits in the application of surgical coaching to daily surgical practice. However, significant knowledge gaps remain. Dedicated research into the short-term applicability and the long-term effects of surgical coaching are required.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy

INTRODUCTION

Self-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.

METHODS AND PROCEDURES

Using a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1-3] and senior trainees (PGY 4-5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.

RESULTS

A total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.

CONCLUSIONS

OPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.

A Novel Expert Coaching Model in Urology, Aimed at Accelerating the Learning Curve in Robotic Prostatectomy

INTRODUCTION/BACKGROUND

The surgical residency model assumes that upon completion, a surgeon is ready to practice and grow independently. However, many surgeons fail to improve after reaching proficiency, which in certain instances has correlated with worse clinical outcomes. Coaching addresses this problem and furthers surgeons' education post-residency. Currently, surgical coaching programs focus on medical students and residents, and have been shown to improve residents' and medical students' technical and non-technical abilities. Coaching programs also increase the accuracy of residents, fellows, and attendings in self-assessing their surgical ability. Despite the potential benefits, coaching remains underutilized and poorly studied. We developed an expert-led, face-to-face, video-based surgical coaching program at a tertiary medical center among specialized attending surgeons. Our goal was to evaluate the feasibility of such a program, measure surgeons' attitudes towards internal peer coaching, determine whether surgeons found the sessions valuable and educational, and to subjectively self-assess changes in operative technique.

METHODS/MATERIALS

Surgeons who perform robot-assisted laparoscopic prostatectomies were chosen and grouped by number of cases completed: junior (<100 cases), intermediate (100-500 cases), and senior (>500 cases). Surgeons were scheduled for 3 1-hour coaching sessions 1-2 months apart (February-October 2019), meeting individually with the coach (PS), an expert Urologic Oncologist with thousands of cases of experience performing radical prostatectomy. He received training on coaching methodology prior to beginning the coaching program. Before each session, surgeons selected 1 of their recent intraoperative videos to review. During sessions, the coach led discussion on topics chosen by the surgeon (i.e. neurovascular bundle dissection, apical dissection, bladder neck); together, they developed goals to achieve before the next session. Subsequent sessions included presentation and discussion of a case occurring subsequent to the prior session. Sessions were coded by discussion topics and analyzed based on level of experience. Surgeons completed a survey evaluating the experience.

RESULTS

All 6 surgeons completed 3 sessions. Five surgeons completed the survey; most respondents evaluated themselves as having improved in desired areas and feeling more confident performing the discussed steps of the operation. Discussed surgical principles varied by experience group; when subjectively quantifying the difficulty of surgical steps, the more difficult steps were discussed by the higher experience groups compared to the junior surgeons. The senior surgeons also focused more on oncologic potency, continence outcomes, and more theory-driven questions while the junior surgeons tended to focus more on anatomic and technique-based questions such as tissue handling and the use of cautery and clips. Overall, the surgeons thought this program provoked critical discussion and subsequently modified their technique, and "agreed" or "strongly agreed" that they would seek further sessions.

CONCLUSIONS

Surgical coaching at a large medical center is not only feasible but was rated positively by surgeons across all levels of experience. Coaching led to subjective self-improvement and increased self-confidence among most surgeons. Surgeons also felt that this program offered a safe space to acquire new skills and think critically after finishing residency/fellowship. Themes discussed and takeaways from the sessions varied based on surgeon experience level. While further research is needed to more objectively quantify the impact coaching has on surgeon metrics and patient outcomes, the results of this study supports the initial "proof-of-concept" of peer-based surgical coaching and its potential benefits in accelerating the learning curve for surgeons' post-residency.

Coaching for Surgeons: A Scoping Review of the Quantitative Evidence

To characterize quantitative studies on coaching interventions for professional surgeons to understand how surgical coaching is defined; examine how different coaching programs are designed, implemented, and evaluated; and identify any relevant research gaps.

Mini Abstract: This scoping review revealed that very few studies have quantitatively assessed coaching programs for surgeons focus on technical and nontechnical skills. Studies demonstrate that coaching is well accepted by surgeons. However, effects on technical/nontechnical skills are inconsistent and no evidence confirms that coaching improves patient outcomes. Additional randomized control trials are needed to evaluate the effects of surgical coaching on surgeons’ performance, patient safety, and surgeons’ wellness.

Video Coaching: A National Survey of Surgical Residency Program Directors

OBJECTIVE

Video coaching has been demonstrated to improve resident and attending skills and is overwhelmingly well received by the participants. However, misperceptions about its utility among those who do not use video coaching may be a barrier to widespread implementation.

DESIGN

Cross-sectional web-based survey SETTING: National survey PARTICIPANTS: Surgical program director members of the Association of Program Directors of Surgery STUDY DESIGN: The survey was developed via a deductive approach after a literature review and was piloted with surgical attendings and residents. All Likert scale were averaged and comparisons between groups was performed via independent t-tests.

RESULTS

There were 52 responses from PDs. 27/52(51.9%) PDs reported their program supported video coaching of residents. PDs from residences with video coaching programs were more likely to believe that video coaching was useful in identifying their own strengths and weakness (p = 0.005), was a useful adjunct for resident feedback (p = 0.024), and a personal library of video recordings would be helpful (p = 0.015) when compared to PDs from residencies without video coaching. Programs without video coaching were more likely to believe barriers to implementation included it being ineffective (p = 0.024) and that the technology was unavailable (p = 0.006). Over 50% of respondents from both groups believed expense, difficulty with set up, time required, and patient privacy were "Very" or "Extremely" likely to be barriers to implementation.

CONCLUSIONS

This is the first national survey of PDs regarding the use of video coaching. Residency programs without video coaching may underestimate the utility of video coaching in training surgical residents.

Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation

OBJECTIVE

This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation.

DESIGN

A search was performed on Scopus and PubMed for the terms "operation," "operating room," "surgery," "resident," "house staff," "graduate medical education," "teaching," "coaching," "assessment," "reflection," "camera," and "video" on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review.

SETTING/PARTICIPANTS

Original, peer-reviewed studies focused on VBC or assessment.

RESULTS

VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room.

CONCLUSIONS

VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.

Coaching in Surgical Education: A Systematic Review

OBJECTIVE

The objectives of this study were to review the coaching literature to (1) characterize the criteria integral to the coaching process, specifically in surgery, and (2) describe how these criteria have been variably implemented in published studies.

BACKGROUND

Coaching is a distinct educational intervention, but within surgery the term is frequently used interchangeably with other more established terms such as teaching and mentoring.

METHODS

A systematic search was performed of the MEDLINE and Cochrane databases to identify studies that used coach/coaching as an intervention for surgeons for either technical or nontechnical skills. Study quality was evaluated using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

A total of 2280 articles were identified and after screening by title, abstract and full text, 35 remained. Thirteen coaching criteria (a-m) were identified in 4 general categories: 1. overarching goal (a. refine performance of an existing skill set), 2. the coach (b. trusting partnership, c. avoids assessment, d. 2-way communication), the coachee (e. voluntary participation, f. self-reflection, g. goal setting, h. action plan, i. outcome evaluation), and the coach-coachee rapport (j. coaching training, k. structured coaching model, l. non-directive, m. open ended questions). Adherence to these criteria ranged from as high of 73% of studies (voluntary participation of coach and coachee) to as low as 7% (use of open-ended questions).

CONCLUSIONS

Coaching is being used inconsistently within the surgical education literature. Our hope is that with establishing criteria for coaching, future studies will implement this intervention more consistently and allow for better comparison and generalization of results.

Association of a Statewide Surgical Coaching Program With Clinical Outcomes and Surgeon Perceptions

OBJECTIVE

To assess risk-adjusted outcomes and participant perceptions following a statewide coaching program for bariatric surgeons.

SUMMARY OF BACKGROUND DATA

Coaching has emerged as a new approach for improving individual surgeon performance, but lacks evidence linking to clinical outcomes.

METHODS

This program took place between October 2015 and February 2018 in the Michigan Bariatric Surgery Collaborative. Surgeons were categorized as coach, participant, or nonparticipant for an interrupted time series analysis. Multilevel logistic regression models included patient characteristics, time trends, and number of sessions. Risk-adjusted overall and surgical complication rates are reported, as are within-group relative risk ratios and 95% confidence intervals. We also compared operative times and report risk differences and 95% confidence intervals. Iterative thematic analysis of semi-structured interviews examined participant and coach perceptions of the program.

RESULTS

The coaching program was viewed favorably by most surgeons and many participants described numerous technical and nontechnical practice changes. The program was not associated with significant change in risk-adjusted complications with relative risks for coaches, participants, and nonparticipants of 0.99 (0.62-1.37), 0.91 (0.64-1.17), and 1.15 (0.83-1.47), respectively. Operative times did improve for participants, but not coaches or nonparticipants, with risk differences of -14.0 (-22.3, -5.7), -1.0 (-4.5, 2.4), and -2.6 (-6.9, 1.7). Future coaching programmatic design should consider dose-complexity matching, hierarchical leveling, and optimizing video review.

CONCLUSIONS

This statewide surgical coaching program was perceived as valuable and surgeons reported numerous practice changes. Operative times improved, but there was no significant improvement in risk-adjusted outcomes.