Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg. 2013;257:476-482. [PMID: 23386240 DOI: 10.1097/sla.0b013e318275b72a]

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Importance

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

Objective

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

Design, Setting, and Participants

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

Exposure

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

Main Outcomes and Measures

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

Results

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

Conclusions and Relevance

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Incentive policy for the comprehensive development of young medical talents: an evolutionary game study

Introduction

Currently in China, there is a lack of well-defined and viable incentive mechanisms at the governmental and hospital levels to support the development of young medical talents, thereby hindering their growth Existing studies primarily investigate the current state and trajectory of incentives, yet they inadequately address the distinctive characteristics of various stakeholders involved in medical talent incentive processes, particularly the lack of research on incentive mechanisms with Chinese attributes.

Methods

This study adopts evolutionary game theory to investigate the dynamics of replication and the strategies for achieving evolutionary stability in the comprehensive development of young medical talents, considering both scenarios with and without supportive policies.

Results

In the absence of any supportive policy measures, the evolutionarily stable strategy (ESS) point is O(0,0), the unstable equilibrium point is C(1,1), and the saddle points are A(0,1), B(1,0). The initial state of the system is at the unstable equilibrium point C(1,1), which means that the young medical talents and medical institutions adopt a combination of strategies (actively seeking comprehensive development and taking incentive measures). Under the scenario with supportive policies, the ESS point is C(1,1), the unstable equilibrium point is O(0,0), and the saddle points are A(0,1), B(1,0). The initial state of the system is at the unstable equilibrium point O(0,0), which means that young medical talents and medical institutions adopt (N,N) strategy combinations (inactively seeking comprehensive development, implementing no incentive measure).

Discussion

(1) Government incentives play a crucial role in motivating young medical talents to seek comprehensive development. (2) The level of government incentive support for young medical talents should exceed the cost increment of individual efforts. Additionally, the policy support provided by the government to medical institutions should surpass the incentive support offered by these institutions to young medical talents. This will enhance the motivation and encouragement efforts of medical institutions in actively promoting comprehensive development among young medical talents. (3) With the backing of certain government incentive policies, medical institutions implementing incentive measures and young medical talents actively seeking comprehensive development will establish a virtuous cycle of mutual promotion.

Nationwide standardization of minimally invasive right hemicolectomy for colon cancer and development and validation of a video-based competency assessment tool (the Right study)

BACKGROUND

Substantial variation exists when performing a minimally invasive right hemicolectomy (MIRH) due to disparities in training, expertise and differences in implementation of innovations. This study aimed to achieve national consensus on an optimal and standardized MIRH technique for colon cancer and to develop and validate a video-based competency assessment tool (CAT) for MIRH.

METHOD

Statements covering all elements of MIRH were formulated. Subsequently, the Delphi technique was used to reach consensus on a standardized MIRH among 76 colorectal surgeons from 43 different centres. A CAT was developed based on the Delphi results. Nine surgeons assessed the same 12 unedited full-length videos using the CAT, allowing evaluation of the intraclass correlation coefficient (ICC).

RESULTS

After three Delphi rounds, consensus (≥80% agreement) was achieved on 23 of the 24 statements. Consensus statements included the use of low intra-abdominal pressure, detailed anatomical outline how to perform complete mesocolic excision with central vascular ligation, the creation of an intracorporeal anastomosis, and specimen extraction through a Pfannenstiel incision using a wound protector. The CAT included seven consecutive steps to measure competency of the MIRH and showed high consistency among surgeons with an overall ICC of 0.923.

CONCLUSION

Nationwide consensus on a standardized and optimized technique of MIRH was reached. The CAT developed showed excellent interrater reliability. These achievements are crucial steps to an ongoing nationwide quality improvement project (the Right study).

Smart surgical glasses for orofacial cleft surgery training in high- and low-income countries: A proof of concept

INTRODUCTION

The coronavirus 2019 pandemic urged us to find alternatives for education through remote proctoring and international surgical collaborations among high-, middle-, and low-income countries. Smart surgical glasses are promising for remote surgical education and international surgical collaborations.

AIMS

This study aimed to assess the usability of smart surgical glasses during cleft surgery and explore their potential in remote surgical education and collaboration.

METHODS

Six plastic surgical cases were randomly selected and recorded using the RODS&CONES glasses in 4K (3840 × 2160p). A 23-point questionnaire was sent to one plastic surgeon, one plastic surgery resident, and eight doctors who were not trained to critically appraise the video and audio quality of the smart surgical glasses and their applicability for remote surgical education.

RESULTS

The participants indicated that the smart glasses had several significant advantages over conventional on-site education, such as facilitating a better view of the surgical field and providing possibilities for remote interaction. The audio quality was considered excellent. The main limitations were image stabilization issues and loss of video connection due to weak wireless fidelity.

CONCLUSIONS

All participants appreciated the use of smart glasses for remote education and considered them a promising tool for enhancing the quality of surgical education. The glasses can enable remote assistance and education of local surgical residents and may facilitate sustainable surgical collaborations among high-, middle-, and low-income countries.

Automatic surgical phase recognition-based skill assessment in laparoscopic distal gastrectomy using multicenter videos

BACKGROUND

Gastric surgery involves numerous surgical phases; however, its steps can be clearly defined. Deep learning-based surgical phase recognition can promote stylization of gastric surgery with applications in automatic surgical skill assessment. This study aimed to develop a deep learning-based surgical phase-recognition model using multicenter videos of laparoscopic distal gastrectomy, and examine the feasibility of automatic surgical skill assessment using the developed model.

METHODS

Surgical videos from 20 hospitals were used. Laparoscopic distal gastrectomy was defined and annotated into nine phases and a deep learning-based image classification model was developed for phase recognition. We examined whether the developed model's output, including the number of frames in each phase and the adequacy of the surgical field development during the phase of supra-pancreatic lymphadenectomy, correlated with the manually assigned skill assessment score.

RESULTS

The overall accuracy of phase recognition was 88.8%. Regarding surgical skill assessment based on the number of frames during the phases of lymphadenectomy of the left greater curvature and reconstruction, the number of frames in the high-score group were significantly less than those in the low-score group (829 vs. 1,152, P < 0.01; 1,208 vs. 1,586, P = 0.01, respectively). The output score of the adequacy of the surgical field development, which is the developed model's output, was significantly higher in the high-score group than that in the low-score group (0.975 vs. 0.970, P = 0.04).

CONCLUSION

The developed model had high accuracy in phase-recognition tasks and has the potential for application in automatic surgical skill assessment systems.

Technical assessment in minimally invasive complete mesocolic excision: Is the complete mesocolic excision competency assessment tool valid and reliable?

AIM

The complete mesocolic excision competency assessment tool (CMECAT) is a novel tool designed to assess technical skills in minimally invasive complete mesocolic excision (CME) surgery. The aim of this study was to assess construct validity and reliability of CMECAT in a clinical context.

METHOD

Colorectal surgeons were asked to submit video recorded laparoscopic CME resections for independent assessment of their technical abilities. The videos were grouped by surgeons' training level, and four established CME experts were recruited as CMECAT assessors. Extended reliability analysis (G-theory) was applied to describe assessor agreement.

RESULTS

A total of 19 videos and 72 assessments were included in the analysis. Overall, technical skills assessed by CMECAT improved with increased training level: the experts scored significantly better than the untrained surgeons (3.3 vs. 2.5 points; p < 0.01). On right-sided resections, significantly higher scores were reported with increased training level for all categories and sections, while for left-sided resections, the variance across groups was smaller and significantly higher scores were only reported for oncological safety describing items. Overall, assessor agreement was high (G-coefficient: 0.81).

CONCLUSION

This study confirms that CMECAT can be applied to video recorded CME cases for technical skill assessment. Further, it can reliably assess technical performance in right sided CME surgery, where construct validity has now been established. More videos are required to evaluate its validity on left colonic CME. In the future, we hope CMECAT can improve feedback during CME training, serve as a tool in certification processes and contribute to distinguishing CME from conventional surgery in future research.

Procedure-based assessment implementation in colorectal surgery: a scoping review

BACKGROUND

Competency-based training (CBT) programs use procedure-based assessments (PBAs) to evaluate trainees' abilities to perform specific procedures in clinical settings, similar to Entrustable Professional Activities (EPAs). PBAs help determine trainees' readiness for advanced training levels. However, there is limited evidence on implementing colorectal-specific PBAs in surgical training schemes. This review aims to identify observed and perceived challenges to implementing PBAs in workplace settings.

METHODS

A scoping review following the Joanna Briggs Institute Protocol for Scoping Reviews (JBI-ScR) was conducted. Eligible studies provided evidence on the implementation, feasibility, and challenges of PBAs in colorectal surgery, including various study designs from retrospective to prospective.

RESULTS

Of the 80 screened studies, 75 were excluded based on exclusion criteria. Most of the included studies were conducted in national training institutions in the United Kingdom, assessing 778 colorectal procedures with specific PBAs. The main facilitators of implementing PBAs were structured assessments, focused assessors' training, and electronic forms usage.

CONCLUSION

This review offers insight into the practicality and feasibility of implementing PBAs in colorectal surgery. Identified challenges include the need for adequate assessor training and the time-consuming nature of the assessment. These findings could improve PBA implementation in colorectal surgery and enhance surgical education quality. However, the limited number of studies and existing literature heterogeneity call for more research to identify other gaps.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy

BACKGROUND

Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review.

METHODS

Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT).

RESULTS

Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42).

CONCLUSION

GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Automatic Surgical Skill Assessment System Based on Concordance of Standardized Surgical Field Development Using Artificial Intelligence

Importance

Automatic surgical skill assessment with artificial intelligence (AI) is more objective than manual video review-based skill assessment and can reduce human burden. Standardization of surgical field development is an important aspect of this skill assessment.

Objective

To develop a deep learning model that can recognize the standardized surgical fields in laparoscopic sigmoid colon resection and to evaluate the feasibility of automatic surgical skill assessment based on the concordance of the standardized surgical field development using the proposed deep learning model.

Design, Setting, and Participants

This retrospective diagnostic study used intraoperative videos of laparoscopic colorectal surgery submitted to the Japan Society for Endoscopic Surgery between August 2016 and November 2017. Data were analyzed from April 2020 to September 2022.

Interventions

Videos of surgery performed by expert surgeons with Endoscopic Surgical Skill Qualification System (ESSQS) scores higher than 75 were used to construct a deep learning model able to recognize a standardized surgical field and output its similarity to standardized surgical field development as an AI confidence score (AICS). Other videos were extracted as the validation set.

Main Outcomes and Measures

Videos with scores less than or greater than 2 SDs from the mean were defined as the low- and high-score groups, respectively. The correlation between AICS and ESSQS score and the screening performance using AICS for low- and high-score groups were analyzed.

Results

The sample included 650 intraoperative videos, 60 of which were used for model construction and 60 for validation. The Spearman rank correlation coefficient between the AICS and ESSQS score was 0.81. The receiver operating characteristic (ROC) curves for the screening of the low- and high-score groups were plotted, and the areas under the ROC curve for the low- and high-score group screening were 0.93 and 0.94, respectively.

Conclusions and Relevance

The AICS from the developed model strongly correlated with the ESSQS score, demonstrating the model's feasibility for use as a method of automatic surgical skill assessment. The findings also suggest the feasibility of the proposed model for creating an automated screening system for surgical skills and its potential application to other types of endoscopic procedures.

Experience of practicing obstetrician-gynecologists in a surgical training program in total laparoscopic hysterectomy

BACKGROUND

Advanced surgical techniques, such as total laparoscopic hysterectomy, are often challenging to acquire beyond fellowship training programs for practicing obstetrician-gynecologists. A lack of formative data currently exist for continuing medical education programs, limiting our understanding of how improvement in surgical skills and training programs occur.

OBJECTIVE

This study aimed to investigate how practicing obstetrician-gynecologists acting as trainees experience a program that aims to teach them total laparoscopic hysterectomy, and to assess whether their surgical skills improve according to data from formative assessment tools and qualitative data from open-ended survey questions and in-depth interviews.

STUDY DESIGN

We report a process analysis of formative data collected during a pilot implementation trial of a surgical training program targeting practicing obstetrician-gynecologists. Eleven consultant obstetrician-gynecologists and 4 experienced surgical mentors participated in 4 hospitals in Queensland, Australia. Total laparoscopic hysterectomy was performed in 700 patients over the course of the study. A total laparoscopic hysterectomy surgical mentorship training program of 10 training days with up to 3 total laparoscopic hysterectomy procedures per day was performed. Both the obstetrician-gynecologists and the surgical mentor completed a formative assessment questionnaire analyzing the trainee's performance after each surgical procedure. Mentors were formatively assessed by the Structured Training Trainer Assessment Report (STTAR) and at the completion of the study by the mini-STTAR, a summative assessment of quality of mentorship. Obstetrician-gynecologists, mentors, hospital leaders, and surgical administrative staff participated in qualitative interviews about the training program.

RESULTS

Over time, there was a demonstrated improvement in trainee performance reported by both mentors and trainees in all competency assessment tool domains as the case number increased, with mentors consistently rating trainees' performance higher than the trainees themselves. Most trainees were satisfied with their mentor in all 31 areas during formative assessment, and at the end of the training, structure, attributes, and role modeling were all rated high (average score >4.5; range, 3.79-5.00), whereas training behavior was rated slightly lower at 4.1 (range, 3.79-4.45). Qualitative interviews demonstrated that the trainees found the training to be a beneficial, hands-on experience.

CONCLUSION

Formative assessment clearly documented improvement in surgical skills during a total laparoscopic hysterectomy training program for consultant obstetrician-gynecologists.

Preserving privacy in surgical video analysis using a deep learning classifier to identify out-of-body scenes in endoscopic videos

Surgical video analysis facilitates education and research. However, video recordings of endoscopic surgeries can contain privacy-sensitive information, especially if the endoscopic camera is moved out of the body of patients and out-of-body scenes are recorded. Therefore, identification of out-of-body scenes in endoscopic videos is of major importance to preserve the privacy of patients and operating room staff. This study developed and validated a deep learning model for the identification of out-of-body images in endoscopic videos. The model was trained and evaluated on an internal dataset of 12 different types of laparoscopic and robotic surgeries and was externally validated on two independent multicentric test datasets of laparoscopic gastric bypass and cholecystectomy surgeries. Model performance was evaluated compared to human ground truth annotations measuring the receiver operating characteristic area under the curve (ROC AUC). The internal dataset consisting of 356,267 images from 48 videos and the two multicentric test datasets consisting of 54,385 and 58,349 images from 10 and 20 videos, respectively, were annotated. The model identified out-of-body images with 99.97% ROC AUC on the internal test dataset. Mean ± standard deviation ROC AUC on the multicentric gastric bypass dataset was 99.94 ± 0.07% and 99.71 ± 0.40% on the multicentric cholecystectomy dataset, respectively. The model can reliably identify out-of-body images in endoscopic videos and is publicly shared. This facilitates privacy preservation in surgical video analysis.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

Assessment methods in laparoscopic colorectal surgery: a systematic review of available instruments

BACKGROUND

Laparoscopic surgery has become the golden standard for many procedures, requiring new skills and training methods. The aim of this review is to appraise literature on assessment methods for laparoscopic colorectal procedures and quantify these methods for implementation in surgical training.

MATERIALS AND METHODS

PubMed, Embase and Cochrane Central Register of Controlled Trials databases were searched in October 2022 for studies reporting learning and assessment methods for laparoscopic colorectal surgery. Quality was scored using the Downs and Black checklist. Included articles were categorized in procedure-based assessment methods and non-procedure-based assessment methods. A second distinction was made between capability for formative and/or summative assessment.

RESULTS

In this systematic review, nineteen studies were included. These studies showed large heterogeneity despite categorization. Median quality score was 15 (range 0-26). Fourteen studies were categorized as procedure-based assessment methods (PBA), and five as non-procedure-based assessment methods. Three studies were applicable for summative assessment.

CONCLUSIONS

The results show a considerable diversity in assessment methods with varying quality and suitability. To prevent a sprawl of assessment methods, we argue for selection and development of available high-quality assessment methods. A procedure-based structure combined with an objective assessment scale and possibility for summative assessment should be cornerstones.

An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery

OBJECTIVE

The American Board of Surgery (ABS) sought to investigate the suitability of video-based assessment (VBA) as an adjunct to certification for assessing technical skills.

BACKGROUND

Board certification is based on the successful completion of a residency program coupled with knowledge and reasoning assessments. VBA is a new modality for evaluating operative skills that have been shown to correlate with patient outcomes after surgery.

METHODS

Diplomates of the ABS were initially assessed for background knowledge and interest in VBA. Surgeons were then solicited to participate in the pilot. Three commercially available VBA platforms were identified and used for the pilot assessment. All participants served as reviewers and reviewees for videos. After the interaction, participants were surveyed regarding their experiences and recommendations to the ABS.

RESULTS

To the initial survey, 4853/25,715 diplomates responded. The majority were neither familiar with VBA, nor the tools used for operative assessments. Two hundred seventy-four surgeons actively engaged in the subsequent pilot. One hundred sixty-nine surgeons completed the postpilot survey. Most participants found the process straightforward. Of the participants, 74% felt that the feedback would help their surgical practice. The majority (81%) remain interested in VBA for continuing medical education credits. Using VBA in continuous certification could improve surgeon skills felt by 70%. Two-thirds of participants felt VBA could help identify and remediate underperforming surgeons. Identified barriers to VBA included limitations for open surgery, privacy issues, and technical concerns.

CONCLUSIONS

VBA is promising as an adjunct to the current board certification process and should be further considered by the ABS.

AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?

INTRODUCTION

Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers.

METHODS

A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection.

RESULTS

An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification.

CONCLUSIONS

Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.

Universal meta-competencies of operative performances: a literature review and qualitative synthesis

BACKGROUND

Prioritizing patient health is essential, and given the risk of mortality, surgical techniques should be objectively evaluated. However, there is no comprehensive cross-disciplinary system that evaluates skills across all aspects among surgeons of varying levels. Therefore, this study aimed to uncover universal surgical competencies by decomposing and reconstructing specific descriptions in operative performance assessment tools, as the basis of building automated evaluation system using computer vision and machine learning-based analysis.

METHODS

The study participants were primarily expert surgeons in the gastrointestinal surgery field and the methodology comprised data collection, thematic analysis, and validation. For the data collection, participants identified global operative performance assessment tools according to detailed inclusion and exclusion criteria. Thereafter, thematic analysis was used to conduct detailed analyses of the descriptions in the tools where specific rules were coded, integrated, and discussed to obtain high-level concepts, namely, "Skill meta-competencies." "Skill meta-competencies" was recategorized for data validation and reliability assurance. Nine assessment tools were selected based on participant criteria.

RESULTS

In total, 189 types of skill performances were extracted from the nine tool descriptions and organized into the following five competencies: (1) Tissue handling, (2) Psychomotor skill, (3) Efficiency, (4) Dissection quality, and (5) Exposure quality. The evolutionary importance of these competences' different evaluation targets and purpose over time were assessed; the results showed relatively high reliability, indicating that the categorization was reproducible. The inclusion of basic (tissue handling, psychomotor skill, and efficiency) and advanced (dissection quality and exposure quality) skills in these competencies enhanced the tools' comprehensiveness.

CONCLUSIONS

The competencies identified to help surgeons formalize and implement tacit knowledge of operative performance are highly reproducible. These results can be used to form the basis of an automated skill evaluation system and help surgeons improve the provision of care and training, consequently, improving patient prognosis.

Development of a procedure-specific tool for skill assessment in left- and right-sided laparoscopic complete mesocolic excision

AIM

To (1) develop an assessment tool for laparoscopic complete mesocolic excision (LCME) and (2) report evidence of its content validity.

METHOD

Assessment statements were revealed through (1) semi-structured expert interviews and (2) consensus by the Delphi method, both involving an expert panel of five LCME surgeons. All experts were interviewed and then asked to rate LCME describing statements from 1 (strongly disagree) to 5 (strongly agree). Responses were returned anonymously to the panel until consensus was reached. Statements were directly included as content in the assessment tool if ≥60% of the experts responded "agree" or "strongly agree" (ratings 4 and 5), with the remaining responses being "neither agree nor disagree" (rating 3). Interclass correlation coefficient (ICC) was calculated for expert agreement evaluation. All included statements were subsequently reformulated as tool items and approved by the experts.

RESULTS

Four Delphi rounds were performed to reach consensus. Disagreement was reported for statements describing instrument handling around pancreas; visualisation of landmarks before inferior mesenteric artery ligation; lymphadenectomy around the inferior mesenteric artery, and division of the terminal ileum and transverse colon. ICC in the last Delphi-round was 0.84. The final tool content included 73 statements, converted to 48 right- and 40 left-sided items for LCME assessment.

CONCLUSION

A procedure-specific, video-based tool, named complete mesocolic excision competency assessment tool (CMECAT), has been developed for LCME skill assessment. In the future, we hope it can facilitate assessment of LCME surgeons, resulting in improved patient outcome after colon cancer surgery.

Serbian National Training Programme for minimally invasive colorectal surgery (LapSerb): short-term clinical outcomes of over 1400 colorectal resections

INTRODUCTION

The Serbian National Training Programme for minimally invasive colorectal surgery (LapSerb) was introduced to implement laparoscopic colorectal surgery across Serbia. The programme aimed to accelerate training of established colorectal surgeons through a competency-based programme. This involved knowledge assessment, workshops, live operating, and competency-based assessment of unedited videos. The aim of this study is to report the outcomes of laparoscopic colorectal resection performed by LapSerb certified surgeons.

METHODS

LapSerb prospectively maintained multicentred database was analysed for laparoscopic colorectal resections from January 2015 to February 2021. Data collected included patient demographics, indications for surgery, perioperative data, and 30-day outcomes.

RESULTS

A total of 1456 laparoscopic colectomies by 24 certified surgeons were included in the final analysis. Mean age was 67 (± 12) years old and male to female ratio was 1:1.5. 83.1% of the colectomies were malignant, mainly due to adenocarcinoma. Anterior resection was the most common procedure with 699 (48%) cases, followed by right and left colectomies with 357 (24.5%) and 303 (21%) procedure respectively. 4.8% of patients required conversion to open surgery. Thirty-day readmission and reoperation rates were 2.3% and 4.7%, respectively. Overall mortality in all cases was 1.1% and R0 resections were achieved in 97.8% of malignant colectomies.

CONCLUSION

The LapSerb programme successfully and safely established laparoscopic colorectal surgery across the country with comparable and acceptable short-term clinical outcomes.

Validation of a Newly Developed Competency Assessment Tool for the Posterior Sagittal Anorectoplasty

INTRODUCTION

 The correction of an anorectal malformation (ARM) is complex and relatively infrequent. Simulation training and subsequent assessment may result in better clinical outcomes. Assessment can be done using a competency assessment tool (CAT). This study aims to develop and validate a CAT for the posterior sagittal anorectoplasty (PSARP) on a simulation model.

MATERIALS AND METHODS

 The CAT-PSARP was developed after consultation with experts in the field. The PSARP was divided into five steps, while tissue and instrument handling were scored separately. Participants of pediatric colorectal hands-on courses in 2019 and 2020 were asked to participate. They performed one PSARP procedure on an ARM simulation model, while being assessed by two objective observers using the CAT-PSARP.

RESULTS

 A total of 82 participants were enrolled. A fair interobserver agreement was found for general skills (intraclass correlation coefficient [ICC] = 0.524, p < 0.001), a good agreement for specific skills (ICC = 0.646, p < 0.001), and overall performance (ICC = 0.669, p < 0.001). The experienced group scored higher on all steps (p < 0.001), except for "anoplasty (p = 0.540)," compared with an inexperienced group.

CONCLUSION

 The CAT-PSARP is a suitable objective assessment tool for the overall performance of the included steps of the PSARP for repair of an ARM on a simulation model.

Standardization of ileoanal J-pouch surgery technique: Quality assessment of minimally invasive ileoanal J-pouch surgery videos

BACKGROUND

Ileal pouch anal anastomosis is a complex procedure associated with significant morbidity, with several complications after ileal pouch anal anastomosis surgery leading to pouch failure. The aim of the study is to evaluate the heterogeneity surrounding the technique of ileoanal J-pouch surgery by assessing the safety and quality of published online peer-reviewed surgical videos.

METHODS

Ileal pouch anal anastomosis videos published on peer-reviewed surgical journals and video channels were edited and anonymized to demonstrate specific steps of the surgical procedure: mobilization and division of the rectum, formation of the ileoanal J-pouch reservoir, anastomosis, and lengthening techniques. The anonymized videos were presented to a group of reviewers with expertise in ileal pouch anal anastomosis blinded to the names and affiliations of the surgeons performing the procedure. Primary outcome was the rate of interobserver variability in the assessment of specific technical steps of the ileal pouch anal anastomosis surgery procedure. Secondary outcome was the appropriateness of the use of surgical videos review as an assessment tool for ileal pouch anal anastomosis surgery, measured as rate of reviewers being unable to answer for poor video quality.

RESULTS

In total, 29 video fragments were distributed, and 13 assessors completed a 60-item survey, organized in 7 major domains. The survey completion rate was 93.4%. Out of a total 729 answers, in 23 (3.2%) the reviewers indicated they were unable to comment due to poor video image, and in 48 (6.5%) were unable to comment due to the particular step not being shown in the procedure. The proportion of assessors rating rectal mobilization technically appropriate ranged from 30.7% to 92.3% and from 7.7% to 69.2% for safety. The level of rectal division was considered appropriate in 0 to 53.8% of the videos, whereas the stapling technique used for rectal division was appropriate in 0 to 70% of the videos.

CONCLUSION

Our study assessed published peer-reviewed videos on ileal pouch anal anastomosis surgery and reported heterogeneity in the safety of the demonstrated techniques. Blind assessment of published peer-reviewed ileal pouch anal anastomosis videos reported a high rate of unsafe or inappropriate technique for rectal mobilization and transection in the reviewed videos, with fair interobserver agreement among reviewers. There is a need for consensus on what is considered safe and appropriate in ileal pouch anal anastomosis surgery. Peer review of ileal pouch anal anastomosis surgery videos could facilitate training and accreditation in this complex procedure.

Impact of Endoscopic Surgical Skill Qualification on Laparoscopic Resections for Rectal Cancer in Japan: The EnSSURE Study

Objective

This cohort study investigated short- and long-term postoperative outcomes of laparoscopic procedures for rectal cancer performed with versus without certified surgeons.

Background

In Japan, the Endoscopic Surgical Skill Qualification System (ESSQS) evaluates surgical skills deemed essential for laparoscopic surgery; however, it is unknown whether this certification contributes to procedural safety.

Methods

Outcomes of laparoscopic rectal resections for cStage II and III rectal cancer performed from 2014 to 2016 at 56 Japanese hospitals were retrospectively reviewed. The impact of having versus not having certified surgeons on postoperative complications and other short- and long-term outcomes were assessed. In cases with ESSQS-certified surgeons, surgeons attended surgery in the capacity of an operator, assistant, scope operator, or advisor.

Results

Overall, 3188 procedures were analyzed, with 2644 procedures performed with and 544 without ESSQS-certified surgeons. A multivariate logistic regression model showed that the adjusted odds ratio of postoperative complications after procedures performed with ESSQS-certified surgeons was 0.68 (95% confidence interval, 0.51-0.91; P = 0.009). The adjusted odds ratios for conversion and pathological R0 resection rates with ESSQS-certified surgeons were 0.20 (P < 0.001) and 2.10 (P = 0.04), respectively. Multiple linear regression analyses showed significantly shorter surgical duration and more harvested lymph nodes for operations performed with ESSQS-certified surgeons. Multivariate Cox regression showed that the adjusted hazard ratios for poor overall and recurrence-free survival after operations performed with ESSQS-certified surgeons were 0.88 (P = 0.35) and 1.04 (P = 0.71), respectively.

Conclusions

This study showed the superiority of the short-term postoperative results for laparoscopic rectal procedures performed with ESSQS-certified surgeons.

Artificial Intelligence-Based Total Mesorectal Excision Plane Navigation in Laparoscopic Colorectal Surgery

BACKGROUND

Total mesorectal excision is the standard surgical procedure for rectal cancer because it is associated with low local recurrence rates. To the best of our knowledge, this is the first study to use an image-guided navigation system with total mesorectal excision.

IMPACT OF INNOVATION

The impact of innovation is the development of a deep learning-based image-guided navigation system for areolar tissue in the total mesorectal excision plane. Such a system might be helpful to surgeons because areolar tissue can be used as a landmark for the appropriate dissection plane.

TECHNOLOGY, MATERIALS, AND METHODS

This was a single-center experimental feasibility study involving 32 randomly selected patients who had undergone laparoscopic left-sided colorectal resection between 2015 and 2019. Deep learning-based semantic segmentation of areolar tissue in the total mesorectal excision plane was performed. Intraoperative images capturing the total mesorectal excision scene extracted from left colorectal laparoscopic resection videos were used as training data for the deep learning model. Six hundred annotation images were created from 32 videos, with 528 images in the training and 72 images in the test data sets. The experimental feasibility study was conducted at the Department of Colorectal Surgery, National Cancer Center Hospital East, Chiba, Japan. Dice coefficient was used to evaluate semantic segmentation accuracy for areolar tissue.

PRELIMINARY RESULTS

The developed semantic segmentation model helped locate and highlight the areolar tissue area in the total mesorectal excision plane. The accuracy and generalization performance of deep learning models depend mainly on the quantity and quality of the training data. This study had only 600 images; thus, more images for training are necessary to improve the recognition accuracy.

CONCLUSION AND FUTURE DIRECTIONS

We successfully developed a total mesorectal excision plane image-guided navigation system based on an areolar tissue segmentation approach with high accuracy. This may aid surgeons in recognizing the total mesorectal excision plane for dissection.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

Development of a Reliable Surgical Quality Assurance System for 2-stage Esophagectomy in Randomized Controlled Trials

OBJECTIVE

The aim was to develop a reliable surgical quality assurance system for 2-stage esophagectomy. This development was conducted during the pilot phase of the multicenter ROMIO trial, collaborating with international experts.

SUMMARY OF BACKGROUND DATA

There is evidence that the quality of surgical performance in randomized controlled trials influences clinical outcomes, quality of lymphadenectomy and loco-regional recurrence.

METHODS

Standardization of 2-stage esophagectomy was based on structured observations, semi-structured interviews, hierarchical task analysis, and a Delphi consensus process. This standardization provided the structure for the operation manual and video and photographic assessment tools. Reliability was examined using generalizability theory.

RESULTS

Hierarchical task analysis for 2-stage esophagectomy comprised fifty-four steps. Consensus (75%) agreement was reached on thirty-nine steps, whereas fifteen steps had a majority decision. An operation manual and record were created. A thirty five-item video assessment tool was developed that assessed the process (safety and efficiency) and quality of the end product (anatomy exposed and lymphadenectomy performed) of the operation. The quality of the end product section was used as a twenty seven-item photographic assessment tool. Thirty-one videos and fifty-three photographic series were submitted from the ROMIO pilot phase for assessment. The overall G-coefficient for the video assessment tool was 0.744, and for the photographic assessment tool was 0.700.

CONCLUSIONS

A reliable surgical quality assurance system for 2-stage esophagectomy has been developed for surgical oncology randomized controlled trials.

ETHICAL APPROVAL

11/NW/0895 and confirmed locally as appropriate, 12/SW/0161, 16/SW/0098.Trial registration number: ISRCTN59036820, ISRCTN10386621.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review

Background

In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature.

Methods

A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools.

Results

Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings.

Conclusion

The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-021-08914-z.

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Can complex surgical interventions be standardized? Reaching international consensus on posterior sagittal anorectoplasty using a modified-Delphi method

BACKGROUND/PURPOSE

In an effort to standardize educational experience, address future physician shortages, and improve quality of care to patients, many surgical specialties are discussing how to maximize exposure to index cases. One solution being explored is telementoring, which requires a well-developed educational curriculum with intraoperative objectives. The American College of Surgery Telementoring Task Force selected anorectal malformation and posterior sagittal anorectoplasty (PSARP) for the repair of imperforate anus as the initial educational focus for this pilot. The purpose of this study was to obtain international consensus on intraoperative learning objectives for a complex surgical procedure.

METHODS

A multidisciplinary team of medical educators and pediatric surgery experts created an outline of essential curricular content and intraoperative learning objectives for PSARP in three clinical scenarios. Twelve international subject matter experts were identified meeting strict inclusion criteria. Intraoperative checklists were revised using the modified-Delphi process.

RESULTS

After five rounds of modifications to the intraoperative checklists, international consensus was achieved for three different clinical scenarios requiring a PSARP: perineal or vestibular fistula, low prostatic fistula, and bladder neck fistula.

CONCLUSIONS

A modified-Delphi approach was successful in generating guidelines for surgical techniques that can be used to standardize intraoperative teaching and expectations for trainees.

TYPE OF STUDY

Diagnostic study LEVEL OF EVIDENCE: Level V (expert opinion).

Clinical Evaluation of Intraoperative Near Misses in Laparoscopic Rectal Cancer Surgery

OBJECTIVE

To investigate the frequency, nature, and severity of intraoperative adverse near miss events within advanced laparoscopic surgery and report any associated clinical impact.

BACKGROUND

Despite implementation of surgical safety initiatives, the intraoperative period is poorly documented with evidence of underreporting. Near miss analyses are undertaken in high-risk industries but not in surgical practice.

METHODS

Case video and data from 2 laparoscopic total mesorectal excision randomized controlled trials were analyzed (ALaCaRT ACTRN12609000663257, 2D3D ISRCTN59485808). Intraoperative adverse events were identified and categorized using the observational clinical human reliability analysis technique. The EAES classification was applied by 2 blinded assessors. EAES grade 1 events (nonconsequential error, no damage, or need for correction) were considered near misses. Associated clinical impact was assessed with early morbidity and histopathology outcomes.

RESULTS

One hundred seventy-five cases contained 1113 error events. Six hundred ninety-eight (62.7%) were near misses (median 3, IQR 2-5, range 0-15) with excellent inter-rater and test-retest reliability (κ=0.86, 95% CI 0.83-0.89, P < 0.001 and κ=0.88, 95% CI 0.85-0.9, P < 0.001 respectively). Significantly more near misses were seen in patients who developed early complications (4 (3-6) vs. 3 (2-4), P < 0.001). Higher numbers of near misses were seen in patients with more numerous (P = 0.002) and more serious early complications (P = 0.003). Cases containing major intraoperative adverse events contained significantly more near misses (5 (3-7) vs. 3 (2-5), P < 0.001) with a major event observed for every 19.4 near misses.

CONCLUSION

Intraoperative adverse events and near misses can be reliably and objectively captured in advanced laparoscopic surgery. Near misses are commonplace and closely associated with morbidity outcomes.

Development of a surgical competency assessment tool for sentinel lymph node dissection by minimally invasive surgery for endometrial cancer

INTRODUCTION

Sentinel lymph node dissection is widely used in the staging of endometrial cancer. Variation in surgical techniques potentially impacts diagnostic accuracy and oncologic outcomes, and poses barriers to the comparison of outcomes across institutions or clinical trial sites. Standardization of surgical technique and surgical quality assessment tools are critical to the conduct of clinical trials. By identifying mandatory and prohibited steps of sentinel lymph node (SLN) dissection in endometrial cancer, the purpose of this study was to develop and validate a competency assessment tool for use in surgical quality assurance.

METHODS

A Delphi methodology was applied, included 35 expert gynecological oncology surgeons from 16 countries. Interviews identified key steps and tasks which were rated mandatory, optional, or prohibited using questionnaires. Using the surgical steps for which consensus was achieved, a competency assessment tool was developed and subjected to assessments of validity and reliability.

RESULTS

Seventy percent consensus agreement standardized the specific mandatory, optional, and prohibited steps of SLN dissection for endometrial cancer and informed the development of a competency assessment tool. Consensus agreement identified 21 mandatory and three prohibited steps to complete a SLN dissection. The competency assessment tool was used to rate surgical quality in three preselected videos, demonstrating clear separation in the rating of the skill level displayed with mean skills summary scores differing significantly between the three videos (F score=89.4; P<0.001). Internal consistency of the items was high (Cronbach α=0.88).

CONCLUSION

Specific mandatory and prohibited steps of SLN dissection in endometrial cancer have been identified and validated based on consensus among a large number of international experts. A competency assessment tool is now available and can be used for surgeon selection in clinical trials and for ongoing, prospective quality assurance in routine clinical care.

Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery

Importance

Complex surgical interventions are inherently prone to variation yet they are not objectively measured. The reasons for outcome differences following cancer surgery are unclear.

Objective

To quantify surgical skill within advanced laparoscopic procedures and its association with histopathological and clinical outcomes.

Design, Setting, and Participants

This analysis of data and video from the Australasian Laparoscopic Cancer of Rectum (ALaCaRT) and 2-dimensional/3-dimensional (2D3D) multicenter randomized laparoscopic total mesorectal excision trials, which were conducted at 28 centers in Australia, the United Kingdom, and New Zealand, was performed from 2018 to 2019 and included 176 patients with clinical T1 to T3 rectal adenocarcinoma 15 cm or less from the anal verge. Case videos underwent blinded objective analysis using a bespoke performance assessment tool developed with a 62-international expert Delphi exercise and workshop, interview, and pilot phases.

Interventions

Laparoscopic total mesorectal excision undertaken with curative intent by 34 credentialed surgeons.

Main Outcomes and Measures

Histopathological (plane of mesorectal dissection, ALaCaRT composite end point success [mesorectal fascial plane, circumferential margin, ≥1 mm; distal margin, ≥1 mm]) and 30-day morbidity. End points were analyzed using surgeon quartiles defined by tool scores.

Results

The laparoscopic total mesorectal excision performance tool was produced and shown to be reliable and valid for the specialist level (intraclass correlation coefficient, 0.889; 95% CI, 0.832-0.926; P < .001). A substantial variation in tool scores was recorded (range, 25-48). Scores were associated with the number of intraoperative errors, plane of mesorectal dissection, and short-term patient morbidity, including the number and severity of complications. Upper quartile-scoring surgeons obtained excellent results compared with the lower quartile (mesorectal fascial plane: 93% vs 59%; number needed to treat [NNT], 2.9, P = .002; ALaCaRT end point success, 83% vs 58%; NNT, 4; P = .03; 30-day morbidity, 23% vs 50%; NNT, 3.7; P = .03).

Conclusions and Relevance

Intraoperative surgical skill can be objectively and reliably measured in complex cancer interventions. Substantial variation in technical performance among credentialed surgeons is seen and significantly associated with clinical and pathological outcomes.

Patient-specific mental rehearsal with three-dimensional models before low anterior resection: randomized clinical trial

Background

It was hypothesized that preparing for a surgical procedure, taking into account individual patient characteristics, may facilitate the procedure and improve surgical quality. The aim of this study was to compare different case-specific, preoperative mental rehearsal methods before minimally invasive rectal cancer surgery.

Methods

In this RCT, patients were allocated in a 1 : 1 : 1 : 1 ratio to four groups: systematic mental rehearsal (SMR) using MRI scans; SMR and three-dimensional (3D) virtual models; SMR and synthetic 3D printed models; and routine practice (control group). Surgeons operating on all but the control group underwent mental rehearsal with the visual aids, including axial MRI scans of the pelvis, interactive 3D virtual models reconstructed from axial MRIs, and synthetic models, manufactured by 3D printing. Operations were video-recorded and assessed by two experts blinded to allocation using two validated scores, the Competency Assessment Tool (CAT) and Objective Clinical Human Reliability Analysis (OCHRA). The primary outcome of the study was surgical performance, measured by the CAT.

Results

Forty-nine patients were randomized and allocated to the four groups. There were 12 participants in each of the control, MRI and SMR, and virtual and SMR groups, whereas the SMR using physical models and simulation group included 13. No difference was observed between groups in median CAT scores (control 30.50, MRI 34.25, virtual 31.75, physical 34.00; P = 0.748, partial η² <0.001, where _pη² is indicative of effect size) or OCHRA scores (anterior, posterior, right and left lateral planes, transection P>0.200, _pη² =0.052–0.088). Time spent not performing dissection was significantly shorter for the SMR with MRI group than for the control (57.5 versus 42 respectively; P < 0.001, _pη² =0.212).

Conclusion

Mental rehearsal did not affect CAT and OCHRA scores of consultant surgeons. Reference number: ISRCTN 75603704 (https://www.isrctn.com).

Development and implementation of an assessment tool to evaluate technical skills in the insertion of implantable venous access devices, a Prospective Cohort Study

OBJECTIVE

Based on the Competency Assessment Tool, herein we developed an assessment instrument suitable to evaluate the implantation of central intravenous devices.

BACKGROUND

Surgical assessment is based mainly on the subjective impressions of the teacher. Based on the "Competency Assessment Tool" (CAT) developed for the evaluation of technical surgical skills in minimally invasive colorectal resection, we designed an assessment tool suitable to evaluate the implantation of central venous access devices performed by junior surgical trainees.

METHODS

Four major assessments during the different steps of the intervention were used in this evaluation. Each of these tasks was divided into four sub-domains according to surgical skill. In addition to the CAT score, the apprentices' skills were evaluated using a visual assessment that was quantified using an analogue scale (value from 1 to 10). The candidates were classified into junior and senior trainees depending on the number of procedures they had already performed and on their surgical experience.

RESULTS

71 procedures were evaluated during the study period. Seven senior trainees conducted 43 procedures and five junior trainees performed 28 interventions. The senior trainees had significantly higher CAT scores than junior candidates, and the scores fluctuated according to surgical experience, usually reaching their peak after 10 procedures.

CONCLUSIONS

The CAT model is well suited for the assessment of surgical trainees during central venous access device implantation. It enables a close assessment of the learning process and the technical skills of trainees, which helps them improving in a safe, standardized manner.

Laparoscopic Colorectal Surgery Outcomes Improved After National Training Program (LAPCO) for Specialists in England

OBJECTIVE

To examine the impact of The National Training Program for Lapco on the rate of laparoscopic surgery and clinical outcomes of cases performed by Lapco surgeons after completion of training.

SUMMARY OF BACKGROUND DATA

Lapco provided competency-based supervised clinical training for specialist colorectal surgeons in England.

METHODS

We compared the rate of laparoscopic surgery, mortality, and morbidity for colorectal cancer resections by Lapco delegates and non-Lapco surgeons in 3-year periods preceding and following Lapco using difference in differences analysis. The changes in the rate of post-Lapco laparoscopic surgery with the Lapco sign-off competency assessment and in-training global assessment scores were examined using risk-adjusted cumulative sum to determine their predictive clinical validity with predefined competent scores of 3 and 5 respectively.

RESULTS

One hundred eight Lapco delegates performed 4586 elective colo-rectal resections pre-Lapco and 5115 post-Lapco while non-Lapco surgeons performed 72,930 matched cases. Lapco delegates had a 37.8% increase in laparoscopic surgery which was greater than non-Lapco surgeons by 20.9% [95% confidence interval (CI), 18.5-23.3, P < 0.001) with a relative decrease in 30-day mortality by -1.6% (95% CI, -3.4 to -0.2, P = 0.039) and 90-day mortality by -2.3% (95% CI, -4.3 to -0.4, P = 0.018). The change point of risk-adjusted cumulative sum was 3.12 for competency assessment tool and 4.74 for global assessment score whereas laparoscopic rate increased from 44% to 66% and 40% to 56%, respectively.

CONCLUSIONS

Lapco increased the rate of laparoscopic colorectal cancer surgery and reduced mortality and morbidity in England. In-training competency assessment tools predicted clinical performance after training.

A methodology for Dynamic Human Reliability Analysis in Robotic Surgery

Surgery has changed significantly in recent years due to the introduction of advanced technologies, resulting in increased system complexity at the technical, human and organisational levels, which may lead to higher variability of patient outcome due to new error pathways. Current approaches towards a safer surgery are largely based on ex-post analysis of events and process monitoring (e.g. root cause analysis, safety checklists, safety audits). However, adopting a proactive approach enables the prior identification of critical factors and the design of safer sociotechnical systems, thanks to a multi-level (or mesoergnomics) perspective. In this paper, a methodology for performing mesoergonomics analysis of surgical procedures is proposed. It is a methodology for Dynamic Human Reliability Analysis in Robotic Surgery based on a modified version of human error assessment and reduction technique (HEART) integrated with a method for incorporating uncertainties related to the influence of personal and organisational factors on the execution of a surgical procedure. The pilot application involves a robot-assisted radical prostatectomy procedure, and the results reveal that team-related factors have the greatest impact on patient outcome variability.

Accuracy and usefulness in assessing proficiency of the observational clinical human reliability assessment checklist of the open inguinal hernia repair procedure: A cross-sectional study

BACKGROUND

The Observational Clinical Human Reliability Assessment (OCHRA) can be used to score errors during surgical procedures. To construct an OCHRA-checklist, steps, substeps, and hazards of a surgical procedure need to be defined. A step-by-step framework was developed to segment surgical procedures into steps, substeps, and hazards. The first aim of this study was to investigate if the step-by-step framework could be used to construct an accurate Lichtenstein open inguinal hernia repair (LOIHR) stepwise description. The second aim was to investigate if the OCHRA-checklist based on this stepwise description was accurate and useful for surgical training and assessment.

MATERIALS AND METHODS

Ten expert surgeons rated statements regarding the accuracy of the LOIHR stepwise description, the accuracy, and the usefulness of the LOIHR OCHRA-checklist (eight, seven, and six statements, respectively) using a 5-point Likert scale. One-sample Wilcoxon signed-rank test was used to compare the outcomes to the neutral value of 3.

RESULTS

The accuracy of the stepwise description and the accuracy and usefulness of the OCHRA-checklist were rated statistically significantly higher than the neutral value of 3 (median 4.75 [5.00-4.00] with p = .009, median 5.00 [5.00-4.00] with p = .012, median 4.00 [5.00-4.00] with p = .047, respectively). The experts rated the OCHRA-checklist to be useful for the training (5.00 [5.00-4.00], p = .009), and assessment (4.50 [5.00-4.00], p = .010) of surgical residents.

CONCLUSION

This preliminary study showed that the stepwise LOIHR description constructed using the step-by-step framework was found to be accurate. The LOIHR OCHRA-checklist developed using the stepwise description was also accurate, and particularly useful for the training and assessment of proficiency of surgical residents.

The what? How? And Who? Of video based assessment

BACKGROUND

Currently, there is significant variability in the development, implementation and overarching goals of video review for assessment of surgical performance.

METHODS

This paper evaluates the current methods in which video review is used for evaluation of surgical performance and identifies which processes are critical for successful, widespread implementation of video-based assessment.

RESULTS

Despite the advances in video capture technology and growing interest in video-based assessment, there is a notable gap in the implementation and longitudinal use of formative and summative assessment using video.

CONCLUSION

Validity, scalability and discoverability are current but removable barriers to video-based assessment.

Development and validation of a recommended checklist for assessment of surgical videos quality: the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool

INTRODUCTION

There has been a constant increase in the number of published surgical videos with preference for open-access sources, but the proportion of videos undergoing peer-review prior to publication has markedly decreased, raising questions over quality of the educational content presented. The aim of this study was the development and validation of a standard framework for the appraisal of surgical videos submitted for presentation and publication, the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool.

METHODS

An international committee identified items for inclusion in the LAP-VEGaS video assessment tool and finalised the marking score utilising Delphi methodology. The tool was finally validated by anonymous evaluation of selected videos by a group of validators not involved in the tool development.

RESULTS

9 items were included in the LAP-VEGaS video assessment tool, with every item scoring from 0 (item not presented in the video) to 2 (item extensively presented in the video), with a total marking score ranging from 0 to 18. The LAP-VEGaS video assessment tool resulted highly accurate in identifying and selecting videos for acceptance for conference presentation and publication, with high level of internal consistency and generalisability.

CONCLUSIONS

We propose that peer review in adherence to the LAP-VEGaS video assessment tool could enhance the overall quality of published video outputs.

Surgical Quality Assurance in COLOR III: Standardization and Competency Assessment in a Randomized Controlled Trial

OBJECTIVE

The aim of this study was to develop an objective and reliable surgical quality assurance system (SQA) for COLOR III, an international multicenter randomized controlled trial (RCT) comparing transanal total mesorectal excision (TaTME) with laparoscopic approach for rectal cancer.

BACKGROUND OF SUMMARY DATA

SQA influences outcome measures in RCTs such as lymph nodes harvest, in-hospital mortality, and locoregional cancer recurrence. However, levels of SQA are variable.

METHOD

Hierarchical task analysis of TaTME was performed. A 4-round Delphi methodology was applied for standardization of TaTME steps. Semistructured interviews were conducted in round 1 to identify key steps and tasks, which were rated as mandatory, optional, or prohibited in rounds 2 to 4 using questionnaires. Competency assessment tool (CAT) was developed and its content validity was examined by expert surgeons. Twenty unedited videos were assessed to test reliability using generalizability theory.

RESULTS

Eighty-three of 101 surgical tasks identified reached 70% agreement (26 mandatory, 56 optional, and 1 prohibited). An operative guide of standardized TaTME was created. CAT is matrix of 9 steps and 4 performance qualities: exposure, execution, adverse event, and end-product. The overall G-coefficient was 0.883. Inter-rater and interitem reliability were 0.883 and 0.986. To enter COLOR III, 2 unedited TaTME and 1 laparoscopic TME videos were submitted and assessed by 2 independent assessors using CAT.

CONCLUSION

We described an iterative approach to develop an objective SQA within multicenter RCT. This approach provided standardization, the development of reliable and valid CAT, and the criteria for trial entry and monitoring surgical performance during the trial.

Validation of Motion Tracking Software for Evaluation of Surgical Performance in Laparoscopic Cholecystectomy

Motion tracking software for assessing laparoscopic surgical proficiency has been proven to be effective in differentiating between expert and novice performances. However, with several indices that can be generated from the software, there is no set threshold that can be used to benchmark performances. The aim of this study was to identify the best possible algorithm that can be used to benchmark expert, intermediate and novice performances for objective evaluation of psychomotor skills. 12 video recordings of various surgeons were collected in a blinded fashion. Data from our previous study of 6 experts and 23 novices was also included in the analysis to determine thresholds for performance. Video recording were analyzed both by the Kinovea 0.8.15 software and a blinded expert observer using the CAT form. Multiple algorithms were tested to accurately identify expert and novice performances. ½ L + [Formula: see text] A + [Formula: see text] J scoring of path length, average movement and jerk index respectively resulted in identifying 23/24 performances. Comparing the algorithm to CAT assessment yielded in a linear regression coefficient R2 of 0.844. The value of motion tracking software in providing objective clinical evaluation and retrospective analysis is evident. Given the prospective use of this tool the algorithm developed in this study proves to be effective in benchmarking performances for psychomotor skills evaluation.

Factors Predicting Operative Difficulty of Laparoscopic Total Mesorectal Excision

BACKGROUND

Laparoscopic total mesorectal excision is a challenging procedure requiring high-quality surgery for optimal outcomes. Patient, tumor, and pelvic factors are believed to determine difficulty, but previous studies were limited to postoperative data.

OBJECTIVE

This study aimed to report factors predicting laparoscopic total mesorectal excision performance by using objective intraoperative assessment.

DESIGN

Data from a multicenter laparoscopic total mesorectal excision randomized trial (ISRCTN59485808) were reviewed.

SETTING

This study was conducted at 4 centers in the United Kingdom.

PATIENTS AND INTERVENTION

Seventy-one patients underwent elective laparoscopic total mesorectal excision for rectal adenocarcinoma with curative intent: 53% were men, mean age was 69 years, body mass index was 27.7, tumor height was 8.5 cm, 24% underwent neoadjuvant therapy, and 25% had previous surgery.

MAIN OUTCOME MEASURES

Surgical performance was assessed through the identification of intraoperative adverse events by using observational clinical human reliability analysis. Univariate analysis and multivariate binomial regression were performed to establish factors predicting the number of intraoperative errors, surgeon-reported case difficulty, and short-term clinical and histopathological outcomes.

RESULTS

A total of 1331 intraoperative errors were identified from 365 hours of surgery (median, 18 per case; interquartile range, 16-22; and range, 9-49). No patient, tumor, or bony pelvimetry measurement correlated with total or pelvic error count, surgeon-reported case difficulty, cognitive load, operative data, specimen quality, number or severity of 30-day morbidity events and length of stay (all r not exceeding ±0.26, p > 0.05). Mesorectal area was associated with major intraoperative adverse events (OR, 1.09; 95%CI, 1.01-1.16; p = 0.015) and postoperative morbidity (OR, 1.1; 95% CI, 1.01-1.2; p = 0.033). Obese men were subjectively reported as harder cases (24 vs 36 mm, p = 0.042), but no detrimental effects on performance or outcomes were seen.

LIMITATIONS

Our sample size is modest, risking type II errors and overfitting of the statistical models.

CONCLUSION

Patient, tumor, and bony pelvic anatomical characteristics are not seen to influence laparoscopic total mesorectal excision operative difficulty. Mesorectal area is identified as a risk factor for intraoperative and postoperative morbidity. See Video Abstract at http://links.lww.com/DCR/B35. FACTORES QUE PREDICEN LA DIFICULTAD OPERATIVA DE LA ESCISIÓN MESORRECTAL TOTAL LAPAROSCÓPICA: La escisión mesorrectal total laparoscópica es un procedimiento desafiante. Para obtener resultados óptimos, se requiere cirugía de alta calidad. Se cree que, factores como el paciente, el tumor y la pelvis, determinan la dificultad, pero estudios previos solamente se han limitado a datos postoperatorios.Informar de los factores que predicen el resultado de la escisión mesorrectal total laparoscópica, mediante una evaluación intraoperatoria objetiva.Datos de un ensayo multicéntrico y randomizado de escisión mesorrectal total laparoscópica (ISRCTN59485808).Cuatro centros del Reino Unido.Un total de 71 pacientes fueron sometidos a escisión mesorrectal total laparoscópica electiva, para adenocarcinoma rectal con intención curativa. 53% hombres, edad media, índice de masa corporal y altura del tumor 69, 27.7 y 8.5 cm respectivamente, 24% terapia neoadyuvante y 25% cirugía previa.Rendimiento quirúrgico evaluado mediante la identificación de eventos intraoperatorios adversos, mediante el análisis clínico observacional de confiabilidad humana. Se realizaron análisis univariado y la regresión binomial multivariada para establecer factores que predicen el número de errores intraoperatorios, reportes del cirujano sobre la dificultad del caso y los resultados clínicos e histopatológicos a corto plazo.Se identificaron un total de 1,331 errores intraoperatorios en 365 horas de cirugía (media de 18 por caso, IQR 16-22, rango 9-49). Ningún paciente, tumor o medición de pelvimetría pélvica, se correlacionó con la cuenta de errores pélvicos o totales, reporte del cirujano sobre dificultad del caso, carga cognitiva, datos operativos, calidad de la muestra, número o gravedad de eventos de morbilidad de 30 días y duración de la estadía (todos r <± 0.26, p > 0.05). El área mesorrectal se asoció con eventos adversos intraoperatorios importantes (OR, 1.09; IC 95%, 1.01-1.16; p = 0.015) y morbilidad postoperatoria (OR, 1.1; IC 95%, 1.01-1.2; p = 0.033). Como información subjetiva, hombres obesos fueron casos más difíciles (24 mm frente a 36 mm, p = 0.042) pero no se observaron efectos perjudiciales sobre el rendimiento o los resultados.Nuestro tamaño de muestra es un modesto riesgo de errores de tipo II y el sobreajuste de los modelos estadísticos.No se observa que las características anatómicas del paciente, tumor y pelvis ósea influyan en la dificultad operatoria de la escisión mesorrectal laparoscópica total. El área mesorrectal se identifica como un factor de riesgo para la morbilidad intraoperatoria y postoperatoria. Vea el resumen del video en http://links.lww.com/DCR/B35.

Video-based coaching in surgical education: a systematic review and meta-analysis

BACKGROUND

In the era of competency-based surgical education, VBC has gained increased attention and may enhance the efficacy of surgical education. The objective of this systematic review was to summarize the existing evidence of video-based coaching (VBC) and compare VBC to traditional master-apprentice-based surgical education.

METHODS

We performed a systematic review and meta-analysis of randomized controlled trials (RCT) assessing VBC according to the PRISMA and Cochrane guidelines. The MEDLINE, EMBASE, and COCHRANE and Researchgate databases were searched for eligible manuscripts. Standard mean difference (SMD) of performance scoring scales was used to assess the effect of VBC versus traditional training without VBC (control).

RESULTS

Of 627 studies identified, 24 RCTs were eligible and evaluated. The studies included 778 surgical trainees (n = 386 VBC vs. n = 392 control). 13 performance scoring scales were used to assess technical competence; OSATS-GRS was the most common (n = 15). VBC was provided preoperative (n = 11), intraoperative (n = 1), postoperative (n = 10), and perioperative (n = 2). The majority of studies were unstructured, where identified coaching frameworks were PRACTICE (n = 1), GROW (n = 2) and Wisconsin Coaching Framework (n = 1). There was an effect on performance scoring scales in favor of VBC coaching (SMD 0.87, p < 0.001). In subgroup analyses, the residents had a larger relative effect (SMD 1.13; 0.61-1.65, p < 0.001) of VBC compared to medical students (SMD 0.43, 0.06-0.81, p < 0.001). The greatest source of potential bias was absence of blinding of the participants and personnel (n = 20).

CONCLUSION

Video-based coaching increases technical performance of medical students and surgical residents. There exist significant study and intervention heterogeneity that warrants further exploration, showing the need to structure and standardize video-based coaching tools.

In Search of Characterizing Surgical Skill

OBJECTIVE

This paper provides a literature review and detailed discussion of surgical skill terminology. Culminating in a novel model that proposes a set of unique definitions, this review is designed to facilitate shared understanding to study and develop metrics quantifying surgical skill.

DESIGN

Objective surgical skill analysis depends on consistent definitions and shared understanding of terms like performance, expertise, experience, aptitude, ability, competency, and proficiency.

STRUCTURE

Each term is discussed in turn, drawing from existing literature and colloquial uses.

IMPLICATIONS

A new model of definitions is proposed to cement a common and consistent lexicon for future skills analysis, and to quantitatively describe a surgeon's performance throughout their career.

Stepwise Training in Laparoscopic Surgery for Complex Ileocolonic Crohn's Disease: Analysis of 127 Training Episodes

INTRODUCTION

The inflammation encountered in Crohn's disease makes a minimally invasive approach challenging due to a thickened mesentery, fistulas, abscesses, and large phlegmons with high reported rates of conversion and septic complications. Aim of this study was to evaluate the feasibility of a stepwise approach to training in laparoscopic surgery for complex Crohn's disease.

METHODS

Every surgical procedure was divided in 4 different training tasks: access and exposure, bowel mobilization, division of the mesentery, anastomosis. Extensive adhesiolysis and division and repair of fistulae were considered as additional tasks when present. The laparoscopic competence assessment tool was used to evaluate the safety and proficiency of the surgical performance. The primary outcome was the rate of training tasks successfully completed by surgical trainees.

RESULTS

One hundred and twenty seven training episodes were included and 86 were performed by trainees (67.7%). Fistula division was the less commonly performed training task (25%), while mobilisation and anastomosis were performed by the supervised trainee in 90% and 85% of the cases. Safety and proficiency scores were significantly higher for senior trainees compared to junior trainees.

CONCLUSIONS

Laparoscopic surgery for complex Crohn's disease can be safely performed in a supervised setting with acceptable operating time, postoperative length of hospital stay, and 30 day morbidity.

Competency assessment tool for laparoscopic suturing: development and reliability evaluation

BACKGROUND

Laparoscopic suturing can be technically challenging and requires extensive training to achieve competency. To date no specific and objective assessment method for laparoscopic suturing and knot tying is available that can guide training and monitor performance in these complex surgical skills. In this study we aimed to develop a laparoscopic suturing competency assessment tool (LS-CAT) and assess its inter-observer reliability.

METHODS

We developed a bespoke CAT tool for laparoscopic suturing through a structured, mixed methodology approach, overseen by a steering committee with experience in developing surgical assessment tools. A wide Delphi consultation with over twelve experts in laparoscopic surgery guided the development stages of the tool. Following, subjects with different levels of laparoscopic expertise were included to evaluate this tool, using a simulated laparoscopic suturing task which involved placing of two surgical knots. A research assistant video recorded and anonymised each performance. Two blinded expert surgeons assessed the anonymised videos using the developed LS-CAT. The LS-CAT scores of the two experts were compared to assess the inter-observer reliability. Lastly, we compared the subjects' LS-CAT performance scores at the beginning and end of their learning curve.

RESULTS

This study evaluated a novel LS-CAT performance tool, comprising of four tasks. Thirty-six complete videos were analysed and evaluated with the LS-CAT, of which the scores demonstrated excellent inter-observer reliability. Cohen's Kappa analysis revealed good to excellent levels of agreement for almost all tasks of both instrument handling and tissue handling (0.87; 0.77; 0.75; 0.86; 0.85, all with p < 0.001). Subjects performed significantly better at the end of their learning curve compared to their first attempt for all LS-CAT items (all with p < 0.001).

CONCLUSIONS

We developed the LS-CAT, which is a laparoscopic suturing grading matrix, with excellent inter-rater reliability and to discriminate between experience levels. This LS-CAT has a potential for wider use to objectively assess laparoscopic suturing skills.

Validation and educational impact study of the NANEP high-fidelity simulation model for open preperitoneal mesh repair of umbilical hernia

OBJECTIVE

The aim of the study was to develop, validate and analyze the educational impact of a high-fidelity simulation model for open preperitoneal mesh repair of an umbilical hernia. The number of surgical simulators available for training residents is limited. Primary for ethical reasons and secondary for the emerging pay-per-quality policies, practicing-on simulators rather than patients is considered gold standard. Validated full-procedural surgical models will become more and more important in training residents. Such models may assure that evidence-based standards regarding technical aspects of the procedures become integral part of the curriculum. Furthermore, they can be employed as a quality control of residents' skills (Fonseca et al. in J Surg Educ 70:129-137, 2013).

METHODS

In a repeated measures design, medical students, residents in their last year of training and attending surgeons performed an open preperitoneal mesh repair on the NANEP model [NANEP stands for the German acronym Nabelhernien-Netzimplatation-Präperitonal (English: Umbilical hernia mesh implantation preperitoneal)]. Subjects were categorized as "Beginners" (internship students) or "Experts" (residents and surgeons). Content validity was analyzed by criteria of subject-matter-experts. Blinded raters assessed surgical skills by means of the Competency Assessment Tool (CAT) using the online platform "CATLIVE". Differential validity was measured by group differences. Proficiency gain was analyzed by monitoring the learning curve (Gallagher et al. in Ann Surg 241:364-372, 2005). Post-operative examination of the simulators shed light on criterion validity.

RESULTS

The NANEP model-proofed content and construct-valid significant Bonferroni-corrected differences were found between beginners and experts (p < 0.05). Beginners showed a significant learning increase from the first to the second surgery (p < 0.05). Post-operative examination data confirmed criterion validity.

CONCLUSION

The NANEP model is an inexpensive, simple and efficient simulation model. It has highly realistic features, it has been shown to be of high-fidelity, full-procedural and benchtop-model. The NANEP model meets the main needs of surgical educational courses at the beginning of residency.

Identifying the Essential Portions of the Skill Acquisition Process Using Item Response Theory

OBJECTIVE

Item response theory (IRT) was originally developed to make performance assessments more accurate. However, IRT analysis of the intraoperative performance of surgical trainees could help identify the elements that the trainees find difficult during the skill acquisition process. The aim of this study was to identify the essential portions of the skill acquisition process of a surgical procedure using the IRT.

DESIGN

The 24-item assessment checklist was used to evaluate a recorded intra-operative performance of a laparoscopic inguinal hernia repair. The scores were analyzed using IRT to calculate the difficulty and discrimination level of each item.

SETTING

Fifteen institutes.

PARTICIPANTS

Thirty surgical trainees.

RESULTS

A total of 123 assessments were analyzed. The item analysis showed the procedure specific item "traction of peritoneum (difficulty: -0.45; discrimination: 19.37)" and generic items "instrument handling (difficulty: -0.59; discrimination: 3.82)" and "flow of procedure (difficulty: 0.09; discrimination: 3.27)" to be key elements in the skill acquisition process of the procedure.

CONCLUSIONS

Key elements in the skill acquisition process of the procedure were quantitatively identified by applying the IRT analysis. This could lead to the use of IRT in designing and developing a more effective training curriculum.