Surgical resident technical skill self-evaluation: increased precision with training progression

The dynamics of self-monitoring in medicine: Safety, efficiency and clinical implications

.@MedEdDoc et al. delve into how the concepts of 'safety' and 'efficiency' in self‐monitoring can be used to influence clinical practice and #MedEd.

What are the learning objectives in surgical training - a systematic literature review of the surgical competence framework

OBJECTIVE

To map the landscape of contemporary surgical education through a competence framework by conducting a systematic literature review on learning outcomes of surgical education and the instructional methods applied to attain the outcomes.

BACKGROUND

Surgical education has seen a paradigm shift towards competence-based training. However, a gap remains in the literature regarding the specific components of competency taught and the instructional methods employed to achieve these outcomes. This paper aims to bridge this gap by conducting a systematic review on the learning outcomes of surgical education within a competence framework and the instructional methods applied. The primary outcome measure was to elucidate the components of competency emphasized by modern surgical curricula. The secondary outcome measure was to discern the instructional methods proven effective in achieving these competencies.

METHODS

A search was conducted across PubMed, Medline, ProQuest Eric, and Cochrane databases, adhering to PRISMA guidelines, limited to 2017-2021. Keywords included terms related to surgical education and training. Inclusion criteria mandated original empirical studies that described learning outcomes and methods, and targeted both medical students and surgical residents.

RESULTS

Out of 42 studies involving 2097 participants, most concentrated on technical skills within competency-based training, with a lesser emphasis on non-technical competencies. The effect on clinical outcomes was infrequently explored.

CONCLUSION

The shift towards competency in surgical training is evident. However, further studies on its ramifications on clinical outcomes are needed. The transition from technical to clinical competence and the creation of validated assessments are crucial for establishing a foundation for lifelong surgical learning.

Exploring the use of metacognitive monitoring cues following a diagram completion intervention

Studying texts constitutes a significant part of student learning in health professions education. Key to learning from text is the ability to effectively monitor one's own cognitive performance and take appropriate regulatory steps for improvement. Inferential cues generated during a learning experience typically guide this monitoring process. It has been shown that interventions to assist learners in using comprehension cues improve their monitoring accuracy. One such intervention is having learners to complete a diagram. Little is known, however, about how learners use cues to shape their monitoring judgments. In addition, previous research has not examined the difference in cue use between categories of learners, such as good and poor monitors. This study explored the types and patterns of cues used by participants after being subjected to a diagram completion task prior to their prediction of performance (PoP). Participants' thought processes were studied by means of a think-aloud method during diagram completion and the subsequent PoP. Results suggest that relying on comprehension-specific cues may lead to a better PoP. Poor monitors relied on multiple cue types and failed to use available cues appropriately. They gave more incorrect responses and made commission errors in the diagram, which likely led to their overconfidence. Good monitors, on the other hand, utilized cues that are predictive of learning from the diagram completion task and seemed to have relied on comprehension cues for their PoP. However, they tended to be cautious in their judgement, which probably made them underestimate themselves. These observations contribute to the current understanding of the use and effectiveness of diagram completion as a cue-prompt intervention and provide direction for future research in enhancing monitoring accuracy.

Perceptions of Competition-Based Learning After a Brief Experience at a National Surgical Meeting

BACKGROUND

Competition-based learning (CBL) facilitates learning through competitions. At the 2022 & 2023 Annual SAGES meetings, we evaluated a CBL experience (TOP GUN Shootout) developed from a modified version of the previously validated TOP GUN Laparoscopic Skills and Suturing Program. The project sought to evaluate the TOP GUN Shootout's (TGS) ability to enhance participant engagement in pursuit of laparoscopic surgical skills.

METHODS

Participants competed in the TGS. Their scores (time and errors) were recorded for: Fundamentals of Laparoscopic Surgery Peg Pass, Cup Drop Task, and Intracorporeal Suturing. All participants completed a 10-question satisfaction survey on a 7-point Likert scale, with questions assessing 3 domains: (1) capability/confidence in MIS skill performance prior to the competition; (2) applicability and satisfaction with TGS's capacity to develop MIS skills; and (3) interest in seeking additional MIS training and appropriateness of CBL in MIS training. Descriptive statistics were used to evaluate these areas.

RESULTS

Overall, 121 participants completed the TGS, of whom 84 (69%) completed the satisfaction survey. The average age was 32.9 years, 67% were males. On average (+/- SD), participant satisfaction was 5.04 (+/- 2.08) for Domain 1, 6.20 (+/- 1.28) for Domain 2, and 6.58 (+/- .95) for Domain 3.

CONCLUSION

Participants described an overall lack of confidence in their MIS skills prior to the 2022-2023 Annual SAGES conference. Participants felt that this brief CBL experience, aided in the development of their MIS skills. Furthermore, this brief CBL experience may inspire learners to seek out further training of their MIS skills.

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis

BACKGROUND

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments.

OBJECTIVES

We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills.

METHODS

The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study.

RESULTS

We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains.

CONCLUSIONS

Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.

A scoping review of self-monitoring in graduate medical education

BACKGROUND

Physicians and physicians-in-training have repeatedly demonstrated poor accuracy of global self-assessments, which are assessments removed from the context of a specific task, regardless of any intervention. Self-monitoring, an in-the-moment self-awareness of one's performance, offers a promising alternative to global self-assessment. The purpose of this scoping review is to better understand the state of self-monitoring in graduate medical education.

METHODS

We performed a scoping review following Arksey and O'Malley's six steps: identifying a research question, identifying relevant studies, selecting included studies, charting the data, collating and summarising the results and consulting experts. Our search queried Ovid Medline, Web of Science, PsychINFO, Eric and EMBASE databases from 1 January 1999 to 12 October 2022.

RESULTS

The literature search yielded 5363 unique articles. The authors identified 77 articles for inclusion. The search process helped create a framework to identify self-monitoring based on time and context dependence. More than 20 different terms were used to describe self-monitoring, and only 13 studies (17%) provided a definition for the equivalent term. Most research focused on post-performance self-judgements of a procedural skill (n = 31, 42%). Regardless of task, studies focused on self-judgement (n = 66, 86%) and measured the accuracy or impact on performance of self-monitoring (n = 41, 71%). Most self-monitoring was conducted post-task (n = 65, 84%).

CONCLUSION

Self-monitoring is a time- and context-dependent phenomenon that seems promising as a research focus to improve clinical performance of trainees in graduate medical education and beyond. The landscape of current literature on self-monitoring is sparse and heterogeneous, suffering from a lack of theoretical underpinning, inconsistent terminology and insufficiently clear definitions.

Video-based learning of coping strategies for common errors improves laparoscopy training-a randomized study

AIMS

The aim of this study was to investigate whether shifting the focus to solution orientation and developing coping strategies for common errors could increase the efficiency of laparoscopic training and influence learning motivation. The concept of coping has been particularly defined by the psychologist Richard Lazarus [Lazarus and Folkman in Stress, appraisal, and coping, Springer publishing company, New York, 1984]. Based on this model, we examined the use of observational learning with a coping model for its effectiveness as a basic teaching model in laparoscopic training.

METHODS

55 laparoscopically naive medical students learned a standardized laparoscopic knot tying technique with video-based instructions. The control group was only offered a mastery video that showed the ideal technique and was free from mistakes. The intervention group was instructed on active error analysis and watched freely selectable videos of common errors including solution strategies (coping model) in addition to the mastery videos.

RESULTS

There was no statistically significant difference between the intervention and control groups for number of knot tying attempts until proficiency was reached (18.8 ± 5.5 vs. 21.3 ± 6.5, p = 0.142). However, there was a significantly higher fraction of knots achieving technical proficiency in the intervention group after first use of the coping model (0.7 ± 0.1 vs. 0.6 ± 0.2, p = 0.026). Additionally, the proportion of blinded attempts that met the criteria for technical proficiency was significantly higher for the intervention group at 60.9% vs. 38.0% in control group (p = 0.021). The motivational subscore "interest" of the validated score on current motivation (QCM) was significantly higher for the intervention group (p = 0.032), as well as subjective learning benefit (p = 0.002) and error awareness (p < 0.001).

CONCLUSION

Using video-based learning of coping strategies for common errors improves learning motivation and understanding of the technique with a significant difference in its qualitative implementation in laparoscopy training. The ability to think in a solution-oriented, independent way is necessary in surgery in order to recognize and adequately deal with technical difficulties and complications.

Anterior plating technique for distal radius: comparing performance after learning through naive versus deliberate practice

BACKGROUND

Surgical teaching is most often carried out in the operating theatre through mentorship, and the performance of surgical procedures is rarely measured. The objective of this article is to compare the progression in learning curves of junior surgeons trained in the anterior plating technique for the distal radius on a nonbiological model according to three different methods.

METHODS

The materials comprised 12 junior surgeons of level 1 or 2 (as per Tang and Giddins) divided into three groups: control (G1), naive practice (G2), and deliberate practice (G3). The three groups watched a demonstration video of a level 5 expert. The four G1 surgeons (two level 1 and two level 2) saw the video only once, and each inserted five plates. The four G2 surgeons (two level 1 and two level 2) inserted five plates and watched the video before each time. The four G3 surgeons (two level 1 and two level 2) saw the video before the first plate insertion. Before posing the subsequent four plates, the four G3 surgeons watched their own video, and the expert indicated their errors and how to avoid them next time. A 12-criteria OSATS defined on the basis of the 60 videos, each graded from one (min.) to five (max.), was used to measure the objective surgical performance per plating (min. 12; max. 60) and per series of five plate fixations (min. 60, max. 300).

RESULTS

The total average objective performance of G1 was 44.73, of G2 was 50.57 and of G3 was 54.35. Change in objective performance was better for G3 (13.25) than G2 (5) or G1 (3.75). For all groups, the progression in objective performance was better amongst level 1 surgeons (9) than level 2 surgeons (5.6).

CONCLUSION

Surgical teaching is based on mentorship and experience. However, since "see one, practice many, do one" has started to replace "see one, do one, teach one", learning techniques have increasingly relied on procedure simulators. Against this background, few studies have looked at measuring the performance of surgical procedures and improved learning curves. Our results appear to suggest that deliberate practice, when used in addition to mentorship, is the best option for shortening the growth phase of the learning curve and improving performance. Deliberate practice is a learning technique for surgical procedures that is complementary to mentorship and experience, which allows the growth phase of the learning curve to be shortened and the objective performance of junior surgeons to be improved.

The Neurostimulation Appropriateness Consensus Committee (NACC): Recommendations for Surgical Technique for Spinal Cord Stimulation

INTRODUCTION

The field of neurostimulation for the treatment of chronic pain is a rapidly developing area of medicine. Although neurostimulation therapies have advanced significantly as a result of technologic improvements, surgical planning, device placement, and postoperative care are of equal importance to optimize outcomes. This Neurostimulation Appropriateness Consensus Committee (NACC) project intends to provide evidence-based guidance for these often-overlooked areas of neurostimulation practice.

MATERIALS AND METHODS

Authors were chosen based on their clinical expertise, familiarity with the peer-reviewed literature, research productivity, and contributions to the neuromodulation literature. Section leaders supervised literature searches of MEDLINE, BioMed Central, Current Contents Connect, Embase, International Pharmaceutical Abstracts, Web of Science, Google Scholar, and PubMed from the last NACC publication in 2017 to the present. Identified studies were graded using the United States Preventive Services Task Force criteria for evidence and certainty of net benefit. Recommendations are based on evidence strength and consensus when evidence was scant.

RESULTS

This NACC project provides guidance on preoperative assessment, intraoperative techniques, and postoperative management in the form of consensus points with supportive evidence. These results are based on grade of evidence, strength of consensus, and expert opinion.

CONCLUSIONS

The NACC has given guidance for a surgical plan that encompasses the patient journey from the planning stage through the surgical experience and postoperative care. The overall recommendations are designed to improve efficacy and the safety of patients undergoing these neuromodulation procedures and are intended to apply throughout the international community.

Educational Research Report Correlation between Student Self-Assessment and Proctor Evaluation in a Veterinary Surgical Laboratory

Self-assessment has been shown to facilitate learning, goal setting, and professional development. We sought to evaluate whether veterinary students in a surgical curriculum would have self-assessments that differed from proctor evaluations and whether high-performing students would differ from low-performing students in self-assessment characteristics. Student and proctor assessments were compared for 8 weeks of a spay/neuter surgical laboratory taking place in the second year of the curriculum. Eight students were classified as high-performing, and 10 students were classified as low-performing. A quantitative evaluation of the scores and a qualitative assessment of written comments were completed. Proctors assigned higher scores to high-performing students compared to low-performing students, but no difference was observed overall in self-assessment scores assigned by students. When only anesthesia students were evaluated, we found a difference in self-assessment scores for high- versus low-performers, but this was not true for surgery students. Differences between proctor and student assessment scores diminished over time for all students and for anesthesia students, but not for surgery students. High-performing student anesthetists self-assessed and received proctor assessments with higher scores in technical skills. Comments from high-performing students tended to be less reflective and more positive. Low-performing students were more defensive and more likely to use I-statements in their comments. Overall, quantitative analysis did not reveal a difference in self-assessment between high-performers and low-performers; however, specific differences existed in qualitative characteristics, surgery versus anesthesia students, and proctor assessments. The differences in self-assessment between high- and low-performing students suggest areas of further investigation for the use of reflection in education.

Performance of Residents During Laparoscopic Cholecystectomy: Is Self-assessment Reliable?

BACKGROUND

To compare the self-assessment of surgical residents and observers (faculty members and nurses) during laparoscopic cholecystectomy (LC).

MATERIALS AND METHODS

A prospective observational study was conducted between February 2020 and July 2020 at a medical school hospital. Seventy-four LC surgeries were performed by surgical residents in the presence of faculty members. A self-assessment of the technical and nontechnical performance of the residents was requested. The self-assessment of residents was compared with observer evaluations using the Kruskal-Wallis test. Gwet AC2 fit coefficient was used to determine the consistency between the observers' and residents' assessments. Bland-Altman plots were generated with 95% limits of agreement to describe the agreement between the total scores of the observers.

RESULTS

The self-assessment of residents had a statistically significant higher score when compared with observers (faculty and nurses) (P<0.001). However, no significant difference was observed between the total scores given by the observers (faculty members and nurses) (P>0.05). There was a moderate agreement between the resident versus faculty members [0.503; 95% confidence interval (CI), 0.430-0.576] and resident versus nurse (0.518; 95% CI, 0.432-0.605) when evaluating technical skills. However, there was substantial agreement between faculty members and nurses (0.736; 95% CI, 0.684-0.789). Postoperative pain was significantly correlated with resident self-assessment (P=0.022).

CONCLUSION

The self-assessment scores of surgical residents in LC operations were overestimated compared with observer assessments.

Impacto de la participación del residente y su rol en desenlaces clínicos de los pacientes llevados a colecistectomía en un hospital universitario

Introducción. La colecistectomía es uno de los procedimientos quirúrgicos más realizados a nivel mundial, por lo que su aprendizaje es cada vez más necesario para los médicos residentes en entrenamiento, pero sin comprometer la seguridad de los pacientes. El objetivo de este estudio fue determinar el impacto de la participación de los médicos residentes en los principales desenlaces clínicos de la colecistectomía. Métodos. Se realizó un estudio prospectivo de cohortes, donde se incluyeron los pacientes llevados a colecistectomía laparoscópica, desde junio de 2019 hasta julio de 2020. Se llevó a cabo el análisis estadístico para describir medidas de frecuencia, tendencia central, dispersión y análisis bivariados para los desenlaces de interés. Resultados. Se incluyeron 482 pacientes a quienes se les practicó colecistectomía, 475 de ellas por vía laparoscópica. El 62,5 % fueron mujeres y el 76,2 % se realizaron de carácter urgente. En el 96 % de los procedimientos se contó con la participación de un residente. En el análisis bivariado no se encontró una diferencia estadísticamente significativa entre la participación del residente y un impacto negativo en los desenlaces clínicos de las variables relevantes. Discusión. No hay evidencia de que la participación de médicos residentes en la colecistectomía laparoscópica se asocie con desenlaces adversos en los pacientes, lo que sugiere estar en relación con una introducción temprana y responsable a este procedimiento por parte de los docentes, permitiendo que la colecistectomía sea un procedimiento seguro.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

Evaluation of a Novel Laparoscopic Cholecystectomy Curriculum With the Use of Animal Models and Live Operating

BACKGROUND

Despite wide recognition of the benefits of simulation training, there is no established model for the teaching of laparoscopic cholecystectomy. The authors developed a replicable, intensive 2-day proficiency-based curriculum, to include simulation and live operating, designed to be practical to both attend and organize. The primary objective of this study was to evaluate this curriculum for improved procedural confidence, measured by participant self-evaluation. Secondary outcomes were objective improvement in technical skills in accordance with the Objective Structured Assessment of Technical Skills (OSATS) scale and trainee self-assessment scores.

METHODS

The course consisted of lectures, operating on a sheep hepatobiliary model through a laparoscopic box trainer, and live operating on female patient volunteers. It was attended by eight junior registrars. Precourse data collated included demographic information, experience, and procedural confidence scores using a visual analog scale. Performance on an animal model and live patient was assessed by experts using the OSATS score. Procedural confidence was re-evaluated after each task, as well as self-assessment of speed, accuracy, and overall performance.

RESULTS

Procedural confidence scores improved by a mean of 12% (P < 0.001). All trainees demonstrated sustained objective improvement in technical skills (P < 0.001). The overall mean OSATS score increased by 18%. Significant improvement was observed after performing the procedure on an animal model (P < 0.001); however, no further significant improvement was observed with live operating. No significant difference was found on trainee self-assessment scores, in any category.

CONCLUSIONS

The study describes a successful curriculum model for the teaching of laparoscopic cholecystectomy, to include procedural and technical skill acquisition, in addition to the refinement and development of procedural confidence. Importantly, this was carried out in a safe environment with direct transferability to the operating theater.

Video-based self-assessment enhances laparoscopic skills on a virtual reality simulator: a randomized controlled trial

BACKGROUND

Hysterectomy rates are decreasing in many countries, and virtual reality (VR) simulators bring new training opportunities for residents. As coaching interventions while training on a simulated complex procedure represents a resource challenge, alternative strategies to improve surgical skills must be investigated. We sought to determine whether self-guided learning using a video-based self-assessment (SA) leads to improved surgical skills in laparoscopic hysterectomy (LH) on a VR simulator.

METHODS

Twenty-four gynecology residents from two university hospitals were randomized into an SA group (n = 12) and a Control group (n = 12). Each participant's baseline performance on a validated VR basic task was assessed. Both groups then performed three virtually simulated LHs during which the participants received no guidance nor feedback. Following each LH, the SA group participants rated the video of their own performance using a generic and a procedure-specific rating scale, while the Control group participants watched an LH video demonstration. The LH videos of both groups' participants were blindly reviewed and rated by expert surgeons, using modified Objective Structured Assessment of Technical Skills scores (OSATS). Objective metrics recorded by the VR simulator were also compared.

RESULTS

There was no difference between the groups' baseline performances on the VR basic task. For the first LH, the OSATS-derived scores did not differ between SA and Control groups (9 [7-13] versus 9 [8-14]; p = 0.728). For the third LH, the OSATS-derived scores were higher for the SA group than for the Control group (17 [15-21] versus 15 [11-17], p = 0.039). Between the two groups, the objective metrics did not differ from the first to the third LH.

CONCLUSIONS

The use of a structured video-based SA leads to improved procedural skills in LH on a VR simulator compared to watching benchmark expert performance, in a population of residents with moderate experience in the operating room.

Cognitive training for robotic surgery: a chance to optimize surgical training? A pilot study

The rapid rise of robotic-assisted surgery (RAS) has necessitated an efficient and standardized training curriculum. Cognitive training (CT) can significantly improve skills, such as attention, working memory and problem solving, and can enhance surgical capacity and support RAS training. This pilot study was carried out between 02/2019 and 04/2019. The participants included 33 student volunteers, randomized into 3 groups: group 1 received training using the da Vinci training simulator, group 2 received computer-based cognitive training, and group 3 was the control group without training. Before (T1) and after-training (T2), performance was measured. Additionally, expert ratings and self-evaluations were collected. Subjective evaluations of performance were supplemented by evaluations based on three scales from the revised NEO Personality Inventory (NEO PI-R). In total, 25 probands remained with complete data for further analyses: n = 8 (group 1), n = 7 (group 2) and n = 10 (group 3). There were no significant differences in T1 and T2 among all three groups. The average training gain of group 1 and 2 was 15.87% and 24.6%, respectively, (a restricting condition is the loss of the last training session in group 2). Analyses of semi-structured psychological interviews (SPIs) revealed no significant differences for T1, but in T2, significance occurred at ‘self-reflection’ for group 2 (F(2.22) = 8.56; p < .005). The efficacy of CT in training highly complex and difficult procedures, such as RAS, is a proven and accepted fact. Further investigation involving higher numbers of training trials (while also being cost effective) should be performed.

Evaluating the Impact of Surgeon Self-Awareness by Comparing Self vs Peer Ratings of Surgical Skill and Outcomes for Bariatric Surgery

OBJECTIVE

To evaluate variation in self vs. peer-assessments of surgical skill using surgical videos and compare surgeon-specific outcomes with bariatric surgery.

SUMMARY BACKGROUND DATA

Prior studies have demonstrated that surgeons with lower peer-reviewed ratings of surgical skill had higher complication rates after bariatric surgery.

METHODS

This is a retrospective cohort study of 25 surgeons who voluntarily submitted a video of a typical laparoscopic sleeve gastrectomy (SG) between 2015-2016. Videos were self and peer-rated using a validated instrument based on a 5-point Likert scale (5="master surgeon" and 1="surgeon-in-training"). Risk adjusted 30-day complication rates were compared between surgeons who over-rated and under-rated their skill based on data from 24,186 SG cases as well as 12,888 gastric bypass (GBP) cases.

RESULTS

Individual overall self-rating of surgical skill varied between 2.5 to 5. Surgeons in the top quartile for self:peer ratings (n=6, ratio 1.58) had lower overall mean peer-scores (2.98 vs 3.79, p = 0.0150) than surgeons in the lowest quartile (n = 6, ratio 0.94). Complication rates between top and bottom quartiles were similar after SG, however leak rates were higher with GBP among surgeons who over-rated their skill with sleeve gastrectomy (0.65 vs 0.27, p = 0.0181). Surgeon experience was similar between comparison groups.

CONCLUSIONS AND RELEVANCE

Self-perceptions of surgical skill varied widely. Surgeons who over-rated their skill had higher leak rates for more complex procedures. Video assessments can help identify surgeons with poor self-awareness who may benefit from a surgical coaching program.

Entrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment

OBJECTIVE

Effective self-assessment is a cornerstone of lifelong professional development; however, evidence suggests physicians have a limited ability to self-assess. Novel strategies to improve the accuracy of learner self-assessment are needed. Our institution's surgical entrustable professional activity (EPA) implementation strategy incorporates resident self-assessment to address this issue. This study evaluates the accuracy of resident self-assessment versus faculty assessment across 5 EPAs in general surgery.

DESIGN, SETTING, PARTICIPANTS

Within a single academic general surgery residency program, assessment data for 5 surgery EPAs was prospectively collected using a mobile application. Matched assessments (resident and faculty assessments for the same clinical encounter) were identified and the remainder excluded. Assessment scores were compared using Welch's t test. Agreement was analyzed using Cohen's kappa with squared weights.

RESULTS

One thousand eight hundred and fifty-seven EPA assessments were collected in 17 months following implementation. One thousand one hundred and fifty-five (62.2%) were matched pairs. Residents under-rated their own performance relative to faculty assessments (2.36 vs 2.65, p < 0.01). This pattern held true for all subsets except for Postgraduate Year (PGY)2 residents and Inguinal Hernia EPAs. There was at least moderate agreement between matched resident and faculty EPA assessment scores (κ = 0.57). This was consistent for all EPAs except Trauma evaluations, which were completed by faculty from 2 different departments. Surgery resident self-assessments more strongly agreed with Surgery faculty assessments than Emergency Medicine faculty assessments (κ = 0.58 vs 0.36).

CONCLUSIONS

Resident EPA self-assessments are equivalent or slightly lower than faculty assessments across a wide breadth of clinical scenarios. Resident and faculty matched assessments demonstrate moderate agreement.

Canadian Plastic Surgery Resident Operative Experience in Hand Surgery

PURPOSE

Core procedural competencies (CPCs) in hand surgery have been previously described. However, it is unknown whether plastic surgery residents receive sufficient operative experience with these procedures. This study aimed to determine whether Canadian plastic surgery residents are receiving adequate exposure to CPCs in hand surgery during residency training.

METHODS

Hand surgery case logs recorded by graduating plastic surgery residents at 10 Canadian English-speaking training programs between 2004 and 2014 were reviewed from 3 databases. For each CPC in hand surgery, perceived resident role and self-competence scores were collected and analyzed. Core procedural competencies encompass core essential procedural competencies (CEPCs) and nonessential procedural competencies.

RESULTS

There were 55 graduating plastic surgery residents included in the study. A total of 14,909 procedures were logged for 42 CPCs, with an average of 271 procedures/resident. When grouped by subdomain within hand surgery, procedures with the most exposure were related to traumatic finger injuries (29%) and neuropathies (22%). The least common procedures were related to flap reconstructions (0.3%) and muscle release (0.1%). Many residents reported receiving no operative experience (range, 2% to 100%) or between 1 and 10 cases of CEPCs (range, 7% to 50%). Most graduating residents (58% to 72%) did not feel competent independently performing 60% of the most common procedures. There were weak to moderate correlations between role and self-competence, and between postgraduate year and self-competence.

CONCLUSIONS

Currently, many plastic surgery residents graduate from training programs with little to no exposure for over half of the CEPCs in hand surgery.

CLINICAL RELEVANCE

Given the great diversity in hand surgery CPCs, it is difficult for residents to gain adequate operative experience during their residency in all of the CPCs.

The effect of verbal feedback, video feedback, and self-assessment on laparoscopic intracorporeal suturing skills in novices: a randomized trial

BACKGROUND

Laparoscopic skill acquisition involves a steep learning curve and laparoscopic suturing is an exceptionally challenging task. By improving the way feedback is given, trainees can learn these skills more effectively. This study aims to establish the most effective form of structured feedback on laparoscopic suturing skill acquisition in novices, by comparing the effects of expert verbal feedback, video review with expert feedback (video feedback), and video review with self-assessment.

METHODS

A prospective randomized blinded trial comparing verbal feedback, video feedback, and self-assessment. Novices in laparoscopic surgery were tasked with performing laparoscopic suturing with intracorporeal knot tying. Time was given for practice, and pre- and post-feedback assessments were undertaken. Suturing performance was measured using a task-specific checklist and global ratings. A post-study questionnaire was used to measure participant-perceived confidence, knowledge, and experience levels.

RESULTS

Fifty-one participants were randomized and allocated equally into the three groups. Performance in all three groups improved significantly from baseline. Video feedback had the largest improvement margin with checklist and global score improvements of 17.1% (± 9.9%) and 14.7% (± 9.3%), respectively. Performance improvements between groups were statistically significant in the global components (p = 0.004) but not the checklist components (p = 0.186). Global score improvement was significantly better in the video feedback group but was statistically insignificant between the self-assessment and verbal feedback groups. Questionnaire responses demonstrated positive results in confidence, knowledge, and experience levels, across all three study groups, with no differences between the groups (p > 0.05).

CONCLUSION

Structured video feedback facilitates reflection and self-directed learning, which improves the ability to develop proficiency in surgical skills. Combining both self-assessment and video feedback may be beneficial over verbal feedback alone due to the advantages of video review. These techniques should therefore be considered for implementation into surgical education curricula.

Differences in operative self-assessment between male and female plastic surgery residents: A survey of 8,149 cases

BACKGROUND

Previous studies show female residents tend to underrate and male residents to overrate their own performance. We sought to determine if plastic surgery resident trainee self-evaluations differ by resident sex.

METHODS

We extracted Operative Entrustability Assessment (OEA) data for plastic surgery programs from MileMarker™, a program capable of storing assessment data for CPT-coded procedures. Complete OEAs contain a trainee self-assessment and attending surgeon assessment. We used simple statistics and linear regression to assess differences, stratifying by trainee sex and post-graduate year (PGY).

RESULTS

We analyzed 8149 OEAs from 3 training programs representing 64 residents (25% female) and 51 attendings. Compared to attending assessments, both male and female residents significantly underrated their performance during PGY1. However, during PGY2-6 male residents' self-evaluations were significantly higher and female residents' self-evaluations significantly lower than their attending evaluations.

CONCLUSIONS

Results demonstrated female plastic surgery residents underestimated and male residents overestimated their performance. Further studies are needed to determine reasons for these differences.

Self-Assessment of Surgical Skills: A Systematic Review

OBJECTIVE

Self-assessment is fundamental in surgical training to enhance learning in the absence of trainer feedback. The primary objective of this review was to assess the factors that influence accuracy of self-assessment at technical skills across all surgical specialties. The secondary objective was to assess whether there are any innate factors or attributes to predict those that will carry out effective self-assessment.

DESIGN

A systematic review was carried out in accordance with PRISMA guidelines. A search strategy encompassing MEDLINE, EMBASE, ERIC, WHO, and the Cochrane database was conducted to identify studies investigating self-assessment at any surgical task. Quality was assessed using the Newcastle-Ottawa scale. A summary table was created to describe specialty, participants, task, setting, assessment tool, and correlation coefficient between self and expert assessment. The review protocol was registered in PROSPERO.

RESULTS

Of 24,638 citations, 40 met inclusion criteria. In total 1753 participants performed 68 procedures. Twenty-six studies investigated skills in general surgery with the remaining 14 in various other surgical specialties. Accuracy of self-assessment is superior in those with greater experience and age, and with use of retrospective video playback. Accuracy tends to be reflected by overestimation of performance. Stressful environments reduce accuracy. There is limited evidence in the literature regarding predicting traits for those who will carry out accurate self-assessment.

CONCLUSIONS

The ability to perform accurate self-assessment is an important skill in surgical training, with accuracy being influenced by a multitude of factors. The use of self-assessment from retrospective video playback may be of benefit in surgical training curricula to enhance learning of technical skills. Further studies are required to define predictors of good self-assessment, which will strengthen recruitment and mentoring to assist trainee learning.

Pain medicine fellow neuromodulation surgical skill assessment tool: a pilot

OBJECTIVE

Spinal cord stimulation (SCS) is an evidence-based therapy for the treatment of refractory pain. Current American College of Graduate Medical Education requirements for pain medicine fellowship graduation include observation of five neuromodulation cases. Matriculating fellows have varying degrees of technical skill, training, and experience. The purpose of this study was to use an innovative skill-based assessment tool using the validated Zwisch scale to evaluate fellow surgical performance during SCS cases.

DESIGN

Cross-sectional survey SETTING: Mayo Clinic, Rochester, Minnesota.

SUBJECTS

Consultant faculty (10) in the Division of Pain Medicine and pain medicine fellows (5).

METHODS

A list of faculty-driven neuromodulation surgical objectives was formed and a rubric was created that focused on technical approach, imaging, epidural access and wound closure. Consultants who perform surgical cases were instructed to use the Zwisch scale as a guide to evaluate fellow surgical performance during a neuromodulation case. Faculty and fellows were surveyed before and after implementation of the tool to determine their satisfaction with use of the skill assessment tool.

RESULTS

The consultants reported they were more satisfied, consistent, and efficient with feedback provided to the trainee on every aspect of neuromodulation surgical cases. The improvement was most significant during the fellows' intraoperative skill assessment. The fellows indicated increased satisfaction, improved communication, and increased efficiency of feedback when the tool was utilized. The fellows reported greater intraoperative skill assessment and consistency of feedback provided.

CONCLUSIONS

The diverse nature of primary specialty backgrounds observed in pain medicine fellowship training offers a unique opportunity to assess and improve fellow skill and surgical competence in the field of neuromodulation. Both faculty and fellows reported improved satisfaction, consistency and efficiency with feedback provided. Importantly, this pilot project observed that implementation of a skill assessment tool was beneficial for both the faculty and fellow as the feedback received was viewed as strongly beneficial to the educational experience.

Pilot Evaluation of the Impact of a Mission-Based Surgical Training Rotation on the Plastic Surgery Skills and Competencies Development of General Surgery Residents in Rwanda

OBJECTIVE

Increasing subspecialty surgical capacity in Rwanda requires innovative approaches to augment the skills of pre-existing surgical providers. Short-term, high-intensity training programs can be effective for surgical education, however, few studies have investigated the quality of skills acquired through a condensed surgical experience. This study aims to determine the efficacy of a 3-week surgical training rotation (STR) to teach plastic surgery skills to general surgery residents in Rwanda.

DESIGN

A survey-based, prospective observational study of general surgery residents and volunteer plastic surgery educators participating in an Operation Smile STR. Resident self-assessment scores of surgical capabilities at the beginning and end of the rotation were compared to surgeon evaluation of resident performance. Progression of resident performance and change in inter-rater reliability between residents and educators were analyzed. Student's t test with significance at p < 0.05 was used to confirmed statistical significance.

SETTING

This study took place during the Operaiton Smile STR at the Rwinkwavu District Hospital, a primary level hospital in Rwinkwavu, Rwanda.

PARTICIPANTS

All residents (5) and surgeon evaluators (4) who participated in the STR were included in this study. All study participants completed all study surveys.

RESULTS

Residents reported a significant increase in confidence with the majority of procedures performed by the end of the rotation. Surgeons identified significant improvement in all resident skills by 2 weeks (p < 0.05). Resident ability to perform self-assessment improved, as determined by a significant decrease in inter-rater error margin from -0.61 to -0.10 over the course of the rotation (p < 0.01).

CONCLUSIONS

In this pilot study, a 3 week rotation improved surgical competencies and technical skills of general surgery residents learning plastic surgery. These findings support using the short-term STR as a method for task-sharing education. Further studies are needed to determine durability of skills transfer and long-term impact on surgical capacity.

The Resident Report Card: A Tool for Operative Feedback and Evaluation of Technical Skills

BACKGROUND

Competency-based medical education surgical curriculums will require frequent, recorded trainee performance evaluations. It is our hypothesis that written feedback after each operation can be used to chart surgical progress, can identify underperforming trainees, and will prove beneficial for resident learning.

METHODS

The resident report card (RRC) is an online, easy-to-use evaluation tool designed to facilitate the creation and distribution of resident technical assessments. RRC data were collected from urologic trainees and analyzed using ANOVA and post hoc testing to confirm our hypothesis. A standardized survey was sent to residents, gauging their views on the RRC.

RESULTS

Over a 5-y period, 958 RRCs with the resident listed as the primary operator were collected across 29 different procedures. Resident cohort and individual performance scores stratified by postgraduate year (PGY) were shown to significantly improve when all procedures (cohort, 6.5 ± 1.9 [PGY-1] to 9.1 ± 1.0 [PGY-5]; individual [resident M], 8.8 ± 1.8 [PGY-3] to 9.4 ± 0.7 [PGY-5], P < 0.01) and specific procedures (laparoscopic donor nephrectomy: cohort, 7.3 ± 1.3 [PGY-3] to 8.9 ± 1.0 [PGY-5]; individual [resident I], 7.2 ± 1.3 [PGY-3] to 9.5 ± 0.6 [PGY-5], P < 0.01) were analyzed. Individual residents were able to be compared to their own peer group and to the average scores across all evaluated residents. Surveyed residents were overwhelmingly positive about the RRC.

CONCLUSIONS

The RRC adds further evidence to the fact that standardized, formative, and timely assessment can capture trainee performance over time and against comparator cohorts in an acceptable format to residents and academic training programs.

Evaluating a Solely Mechanical Articulating Laparoscopic Device: A Prospective Randomized Crossover Study

BACKGROUND

The FlexDex^® (FD) is a solely mechanical articulating device that combines the functionality of robotic surgery with the relative low cost and simplicity of laparoscopy. We sought to evaluate the performance of first-time FD users while performing a simple suture task at locations of varying degrees of difficulty.

STUDY DESIGN

A prospective, randomized crossover study was performed comparing the FD to standard laparoscopy (SL). Two specific groups were evaluated; Group 1 consisted of complete novices, and Group 2 consisted of surgical trainees. Participants performed a simple suture with both FD and SL locations of varying degrees of difficulty (Easy, Moderate, and Hard). The following outcomes were evaluated: Instrument Function and Ergonomics (Comfort/Ergonomics survey), Task Difficulty (National Aeronautics and Space Administration Task Load Index [NASA-TLX]), Task Performance Quality (Objective Structured Assessment of Technical Skills [OSATS]), and Time (seconds).

RESULTS

Twenty-two participants were enrolled with 12 participants in Group 1 and 10 participants in Group 2. Group 1-FD participants experienced overall less shoulder strain (1.2 ± 0.40 versus 1.9 ± 0.90, P = .01), and Group 2-FD participants experienced less shoulder (2.5 ± 0.66 versus 4.0 ± 0.50, P = .01), back (1.1 ± 0.32 versus 1.9 ± 0.74, P = .01), and forearm strain (1.9 ± 0.88 versus 2.5 ± 1.1, P = .04). Group 1 participants using the FD experienced higher mental demand (73 ± 17 versus 48 ± 27, P < .01) and perceived effort (70 ± 20 versus 54 ± 23, P < .001). Both Group 1 and Group 2 FD participants performed tasks at the Hard location more effectively. Both Group 1 (70 versus 87, P = .21) and Group 2 (53 versus 60, P = .55) performed tasks at the Hard location in similar times, while Group 1 (80 versus 177, P = .03) and Group 2 (33 versus 70, P = .001) performed tasks at the Easy location in shorter times using SL.

CONCLUSIONS

This study demonstrates the first assessment of the FD, a mechanically articulating laparoscopic tool. First-time FD users demonstrated improved ergonomics and effectiveness suturing at difficult locations. Future studies will focus on comparison to robotic surgery and translation into clinical applications.