Development, validation and implementation of a monitoring tool for training in laparoscopic colorectal surgery in the English National Training Program

Development of a reliable surgical quality assurance tool for gastrectomy in oncological trials

BACKGROUND

Despite its recognized importance, there is currently no reliable tool for surgical quality assurance (SQA) of gastrectomy in surgical oncology. The aim of this study was to develop an SQA tool for gastrectomy and to apply this tool within the ADDICT Trial in order to assess the extent and completeness of lymphadenectomy.

METHODS

The operative steps for D1+ and D2 gastrectomy have been previously described in the literature and ADDICT trial manual. Two researchers also performed fieldwork in the UK and Japan to document key operative steps through photographs and semi-structured interviews with expert surgeons. This provided the steps that were used as the framework for the SQA tool. Sixty-two photographic cases from the ADDICT Trial were rated by three independent surgeons. Generalizability (G) theory determined inter-rater reliability. D-studies examined the effect of varying the number of assessors and photographic series they rated. Chi-square assessed intra-rater reliability, comparing how the individual assessor's responses corresponded to their global rating for extent of lymphadenectomy.

RESULTS

The tool comprised 20 items, including 19 anatomical landmarks and a global rating score. Overall reliability had G-coefficient of 0.557. Internal consistency was measured with a Cronbach's alpha score of 0.869 and Chi-square confirmed intra-rater reliability for each assessor as < 0.05.

CONCLUSIONS

A photographic surgical quality assurance tool is presented for gastrectomy. Using this tool, the assessor can reliably determine not only the quality but also the extent of the lymphadenectomy performed based on remaining anatomy rather than the excised specimen.

Evaluation of the advantage of surgeons certified by the endoscopic surgical skill qualification system participating in laparoscopic low anterior rectal resection

Background

A technical qualification system was developed in 2004 by the Japan Society for Endoscopic Surgery. An analysis of the EnSSURE study on 3188 stage II-III rectal cancer patients, which was performed by including the participation of qualified surgeons as assistants and advisers without restricting their participation as operators, revealed that the participation of technically qualified surgeons in surgery improved the technical and oncological safety of laparoscopic rectal resection.

Aim

This secondary retrospective analysis of the EnSSURE study examined the advantage of qualified surgeons participating in laparoscopic low anterior resection (LAR).

Methods

The outcomes of low anterior resection were compared between groups with and without the participation of surgeons qualified by the Endoscopic Surgical Skill Qualification System (Q and non-Q groups, respectively). We used propensity score matching to generate paired cohorts at a one-to-one ratio. The postoperative complication rate, short-term results (hemorrhage volume, operative time, number of dissected lymph nodes, open conversion rate, intraoperative complication rate, and R0 resection rate), and long-term results (disease-free survival rate, local recurrence rate, and overall survival rate) were evaluated.

Results

The frequencies of postoperative complications, anastomotic bleeding, and intraperitoneal abscess were significantly lower, the operative time was significantly shorter, the postoperative hospital stay was significantly shorter, and the number of dissected lymph nodes was higher in the Q group. No significant differences were observed in disease-free survival, local recurrence, or overall survival rate rates between the groups.

Conclusion

The participation of qualified surgeons in LAR is technically advantageous.

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Importance

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

Objective

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

Design, Setting, and Participants

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

Exposure

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

Main Outcomes and Measures

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

Results

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

Conclusions and Relevance

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Evaluation of a Microsurgery Training Curriculum

BACKGROUND

 Microsurgery is one of the most challenging areas of surgery with a steep learning curve. To address this educational need, microsurgery curricula have been developed and validated, with the majority focus on technical skills only. The aim of this study was to report on the evaluation of a well-established curriculum using the Kirkpatrick model.

METHODS

 A training curriculum was delivered over 5 days between 2017 and 2020 focusing on (1) microscopic field manipulation, (2) knot tying, nondominant hand usage, (3) 3-D models/anastomosis, and (4) tissue experience. The Kirkpatrick model was applied to evaluate the curriculum at four levels: (1) participants' feedback (2) skills development using a validated, objective assessment tool (Global Assessment Score form) and CUSUM charts were constructed to model proficiency gain (3) and (4) assessing skill retention/long-term impact.

RESULTS

 In total, 155 participants undertook the curriculum, totaling 5,425 hours of training. More than 75% of students reported the course as excellent, with the remaining voting for "good." All participants agreed that the curriculum met expectations and would recommend it. Significant improvement in anastomosis attainment scores between days 1 and 3 (median score 4) and days 4 and 5 (median score 5) (W = 494.5, p = 0.00170). The frequency of errors reduced with successive attempts (chi square = 9.81, p = 0.00174). The steepest learning curve was in anastomosis and patency domains, requiring 11 attempts on average to reach proficiency. In total, 88.5% survey respondents could apply the skills learnt and 76.9% applied the skills learnt within 6 months. Key areas of improvement were identified from this evaluation, and actions to address them were implemented in the following programs.

CONCLUSION

 Robust evaluation of curriculum can be applied to microsurgery training demonstrating its efficacy in reducing surgical errors with an improvement in overall technical skills that can extend to impact clinical practice. It allows the identification of areas of improvement, driving the refinement of training programs.

Procedure-based assessment implementation in colorectal surgery: a scoping review

BACKGROUND

Competency-based training (CBT) programs use procedure-based assessments (PBAs) to evaluate trainees' abilities to perform specific procedures in clinical settings, similar to Entrustable Professional Activities (EPAs). PBAs help determine trainees' readiness for advanced training levels. However, there is limited evidence on implementing colorectal-specific PBAs in surgical training schemes. This review aims to identify observed and perceived challenges to implementing PBAs in workplace settings.

METHODS

A scoping review following the Joanna Briggs Institute Protocol for Scoping Reviews (JBI-ScR) was conducted. Eligible studies provided evidence on the implementation, feasibility, and challenges of PBAs in colorectal surgery, including various study designs from retrospective to prospective.

RESULTS

Of the 80 screened studies, 75 were excluded based on exclusion criteria. Most of the included studies were conducted in national training institutions in the United Kingdom, assessing 778 colorectal procedures with specific PBAs. The main facilitators of implementing PBAs were structured assessments, focused assessors' training, and electronic forms usage.

CONCLUSION

This review offers insight into the practicality and feasibility of implementing PBAs in colorectal surgery. Identified challenges include the need for adequate assessor training and the time-consuming nature of the assessment. These findings could improve PBA implementation in colorectal surgery and enhance surgical education quality. However, the limited number of studies and existing literature heterogeneity call for more research to identify other gaps.

Assessment methods in laparoscopic colorectal surgery: a systematic review of available instruments

BACKGROUND

Laparoscopic surgery has become the golden standard for many procedures, requiring new skills and training methods. The aim of this review is to appraise literature on assessment methods for laparoscopic colorectal procedures and quantify these methods for implementation in surgical training.

MATERIALS AND METHODS

PubMed, Embase and Cochrane Central Register of Controlled Trials databases were searched in October 2022 for studies reporting learning and assessment methods for laparoscopic colorectal surgery. Quality was scored using the Downs and Black checklist. Included articles were categorized in procedure-based assessment methods and non-procedure-based assessment methods. A second distinction was made between capability for formative and/or summative assessment.

RESULTS

In this systematic review, nineteen studies were included. These studies showed large heterogeneity despite categorization. Median quality score was 15 (range 0-26). Fourteen studies were categorized as procedure-based assessment methods (PBA), and five as non-procedure-based assessment methods. Three studies were applicable for summative assessment.

CONCLUSIONS

The results show a considerable diversity in assessment methods with varying quality and suitability. To prevent a sprawl of assessment methods, we argue for selection and development of available high-quality assessment methods. A procedure-based structure combined with an objective assessment scale and possibility for summative assessment should be cornerstones.

Development of a formative feedback tool for transanal total mesorectal excision

INTRODUCTION

Transanal total mesorectal excision (TaTME) is a novel procedure in the treatment of rectal cancer. Current training models for TaTME suggest a period of proctored cases, but no structured feedback tool exists to guide operators during the learning phase. The objective of this study therefore was to develop a formative feedback tool for the critical steps of the transanal portion of TaTME.

METHODS

A two-round Delphi study by TaTME experts was conducted to determine the items to be included in the formative feedback tool. Participants rated each step from a prepared list using a Likert scale from 1 (Not relevant) to 5 (Very relevant) with the option to suggest additional steps. Responses to the first round were presented in the second round, where participants rated the revised list of steps. Consensus was defined as > 80% of participants rating the step as 4 or 5 (out of 5). Items were combined when appropriate to avoid redundancy. Rating anchors describing performance (on a 5-point scale) were then developed for each step. The final tool was recirculated and participants rated the finished product on its feasibility and usefulness.

RESULTS

Twenty-six TaTME experts were contacted for participation. Fifteen experts (58%) participated in the first round of the study, and eleven (42%) participated in the second round. The majority (14, 93%) had completed fellowship training in colorectal surgery. The first round of the Delphi study contained 34 items, and 32 items met inclusion criteria after the second round. Redundant items were combined into 15 items that comprised the final tool. Out of eight respondents to the feasibility survey, all believed the feedback tool enhances the feedback of learners and would use it for training purposes if available.

CONCLUSION

This work describes the development of a novel consensus-based formative feedback tool specific to TaTME.

Change point analysis validation of the learning curve in laparoscopic colorectal surgery: Experience from a non-structured training setting

BACKGROUND

The introduction of minimal invasive principles in colorectal surgery was a major breakthrough, resulting in multiple clinical benefits, at the cost, though, of a notably steep learning process. The development of structured nation-wide training programs led to the easier completion of the learning curve; however, these programs are not yet universally available, thus prohibiting the wider adoption of laparoscopic colorectal surgery.

AIM

To display our experience in the learning curve status of laparoscopic colorectal surgery under a non-structured training setting.

METHODS

We analyzed all laparoscopic colorectal procedures performed in the 2012-2019 period under a non-structured training setting. Cumulative sum analysis and change-point analysis (CPA) were introduced.

RESULTS

Overall, 214 patients were included. In terms of operative time, CPA identified the 110^th case as the first turning point. A plateau was reached after the 145^th case. Subgroup analysis estimated the 58^th for colon and 52^nd case for rectum operations as the respective turning points. A learning curve pattern was confirmed for pathology outcomes, but not in the conversion to open surgery and morbidity endpoints.

CONCLUSION

The learning curves in our setting validate the comparability of the results, despite the absence of National or Surgical Society driven training programs.

Supervised training of laparoscopic colorectal cancer resections does not adversely affect short- and long-term outcomes: a Propensity-score-matched cohort study

Background

Supervised training of laparoscopic colorectal cancer surgery to fellows and consultants (trainees) may raise doubts regarding safety and oncological adequacy. This study investigated these concerns by comparing the short- and long-term outcomes of matched supervised training cases to cases performed by the trainer himself.

Methods

A prospective database was analysed retrospectively. All elective laparoscopic colorectal cancer resections in curative intent of adult patients (≥ 18 years) which were performed (non-training cases) or supervised to trainees (training cases) by a single laparoscopic expert surgeon (trainer) were identified. All trainees were specialist surgeons in training for laparoscopic colorectal surgery. Supervised training was standardised. Training cases were 1:1 propensity-score matched to non-training cases using age, American Society of Anesthesiologists (ASA) grade, tumour site (rectum, left and right colon) and American Joint Committee on Cancer (AJCC) tumour stage. The resulting groups were analysed for both short- (operative, oncological, complications) and long-term (time to recurrence, overall and disease-free survival) outcomes.

Results

From 10/2006 to 2/2016, a total of 675 resections met the inclusion criteria, of which 95 were training cases. These resections were matched to 95 non-training cases. None of the matched covariates exhibited an imbalance greater than 0.25 (│d│>0.25). There were no significant differences in short- (length of procedure, conversion rate, blood loss, postoperative complications, R0 resections, lymph node harvest) and long-term outcomes. When comparing training cases to non-training cases, 5-year overall and disease-free survival rates were 71.6% (62.4–82.2) versus 81.9% (74.2–90.4) and 70.0% (60.8–80.6) versus 73.6% (64.9–83.3), respectively (not significant). The corresponding hazard ratios (95% confidence intervals, p) were 0.57 (0.32–1.02, p = 0.057) and 0.87 (0.51–1.48, p = 0.61), respectively (univariate Cox proportional hazard model).

Conclusions

Standardised supervised training of laparoscopic colorectal cancer procedures to specialist surgeons may not adversely impact short- and long-term outcomes. This result may also apply to newer surgical techniques as long as standardised teaching methods are followed.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

OUP accepted manuscript

BACKGROUND

Surgical training is aimed towards entrusted professional activity to obtain operative independence. Laparoscopic appendicectomy is performed early in training but except for simulators, real-life evaluation towards proficiency is scarce. The aim of this study was to model how each consecutive step may impact on the overall proficiency score for surgical trainees performing laparoscopic appendicectomy.

METHODS

This was an observational cohort study of laparoscopic appendicectomy performed by junior trainees (PGY1-4) under supervision and evaluated for each of eight steps. Each step was scored on a validated six-point performance scale and classified as 'fail', 'pass', or 'proficient'. Modelling was conducted with a multivariable regression model and artificial neural network model with a multilayer perceptron for the relationship between steps and overall performance.

RESULTS

Of 157 procedures, 97 (61.8 per cent) procedures were evaluated as 'proficient', 46 (29.3 per cent) were 'pass', and 14 (8.9 per cent) were 'fail'. In regression analyses, handling the mesoappendix was significantly associated with procedure proficiency, as were division of appendix, access to abdomen, and ability to handle the small bowel. The widest variation in operative flow was shown for steps involving mesoappendix and division of appendix, conceptualized in 'ebb-and-flow' and 'string-of-pearls' models. Sensitivity analyses for experience using 20 or fewer, 30 or fewer, or more than 30 procedures as cut-offs reproduced comparable results.

CONCLUSIONS

Consistent stumbling blocks for junior trainees performing laparoscopic appendectomies can be conceptualized through novel models that identify steps deemed to be the most difficult to trainees with variable experience.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review

Background

In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature.

Methods

A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools.

Results

Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings.

Conclusion

The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-021-08914-z.

Deciphering the inflection points to achieve proficiency for each procedure step during training in laparoscopic appendicectomy

Background

Laparoscopic appendicectomy is a common procedure early in surgical training. A minimum number is usually required for certification in general surgery. However, data on proficiency are scarce. This study aimed to investigate steps towards proficiency in laparoscopic appendicectomy.

Methods

This was a prospective observational cohort study of laparoscopic appendicectomies performed by junior trainees under supervision scored on a six-point performance scale. Structured assessment was done within a defined programme. Procedures performed for uncomplicated appendicitis in adults were included. The procedures were evaluated with LOWESS graphs generated to investigate inflection points. Factors associated with proficiency rates were reported with odds ratios and 95 per cent confidence intervals.

Results

In total 142 laparoscopic procedures were included for 19 trainees (58 per cent female). The cumulative number of procedures during the study was a median of 20 (i.q.r. 8–33). For overall proficiency, an inflection point occurred at 30 procedures. Proficiency rate increased from 51 per cent for 30 or fewer procedures to 93 per cent for more than 30 procedures (odds ratio 11.9 (95 per cent c.i. 3.4 to 40.9); P < 0.001). Inflection points for proficiency for each procedure step varied considerably, with lowest numbers (fewer than 15 procedures) for removing the specimen, and highest for dividing the mesoappendix (more than 55 procedures). Operating time was significantly reduced by a median of 7 minutes after 30 procedures, from median 62 (i.q.r. 25–120) minutes to median 55 (i.q.r. 30–110) minutes for more than 30 procedures.

Conclusion

For junior trainees, variation in proficiency is related to specific procedure steps. Targeted training on specific procedure skills may reduce numbers needed to achieve proficiency in laparoscopic appendicectomy during training.

Development and clinical implementation of a structured, simulation-based training programme in laparoscopic appendectomy: description, validation and evaluation

Background

Laparoscopic appendectomy is a common procedure in general surgery but is likely underused in structured and real-life teaching. This study describes the development, validation and evaluation of implementing a structured training programme for laparoscopic appendectomy.

Study design

A structured curriculum and simulation-based programme for trainees and trainers was developed. All general surgery trainees and trainers were involved in laparoscopic appendectomies. All trainees and trainers underwent the structured preprocedure training programme before real-life surgery evaluation. A standardised form evaluated eight technical steps (skills) of the procedure as well as an overall assessment, and nine elements of communication (feedback), and was used for bilateral evaluation by each trainee and trainer. A consecutive, observational cohort over a 12-month period was used to gauge real-life implementation.

Results

During 277 eligible real-life appendectomies, structured evaluation was performed in 173 (62%) laparoscopic appendectomies, for which 165 forms were completed by 19 trainees. Construct validity was found satisfactory. Inter-rater reliability demonstrated good correlation between trainee and trainer. The trainees’ and trainers’ stepwise and overall assessments of technical skills had an overall good reliability (intraclass correlation coefficient of 0.88). The vast majority (92.2%) of the trainees either agreed or strongly agreed that the training met their expectations.

Conclusion

Structured training for general surgery residents can be implemented for laparoscopic appendectomy. Skills assessment by trainees and trainers indicated reliable self-assessment. Overall, the trainees were satisfied with the training, including the feedback from the trainers.

Proctorship in Minimally Invasive Colorectal Surgery

Teaching an established surgeon in a novel technique by a colleague who has acquired a level of expertise is often referred to as "proctoring" or "precepting." Surgical preceptorships can be defined as supervised teaching programs, whereby individual or groups of surgeons (proctors) experienced in a certain technique support a colleague who wants to adopt this technique (sometimes referred to as "delegates" or "preceptees"). Preceptorship programs really focus on a specific technique, technology, or skill which is required to broaden, complement, or transform an established surgeon's practice. Within colorectal surgery, in the past 30 years, there is been an evolution of interventional options including open, laparoscopic, robotic, and endoscopic procedures. With each new emerging technology and technique, safe and effective uptake by established surgeons is best been attained by a period of proctorship by an experienced colleague. Formalizing this has been facilitated largely through industry support. There, however, remains a considerable chasm when it comes to standardization, quality control, and jurisprudence. This article aims to describe the requirements for a contemporary proctorship program, to examine instruments of quality control, and how to improve effectiveness.

Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery

Importance

Complex surgical interventions are inherently prone to variation yet they are not objectively measured. The reasons for outcome differences following cancer surgery are unclear.

Objective

To quantify surgical skill within advanced laparoscopic procedures and its association with histopathological and clinical outcomes.

Design, Setting, and Participants

This analysis of data and video from the Australasian Laparoscopic Cancer of Rectum (ALaCaRT) and 2-dimensional/3-dimensional (2D3D) multicenter randomized laparoscopic total mesorectal excision trials, which were conducted at 28 centers in Australia, the United Kingdom, and New Zealand, was performed from 2018 to 2019 and included 176 patients with clinical T1 to T3 rectal adenocarcinoma 15 cm or less from the anal verge. Case videos underwent blinded objective analysis using a bespoke performance assessment tool developed with a 62-international expert Delphi exercise and workshop, interview, and pilot phases.

Interventions

Laparoscopic total mesorectal excision undertaken with curative intent by 34 credentialed surgeons.

Main Outcomes and Measures

Histopathological (plane of mesorectal dissection, ALaCaRT composite end point success [mesorectal fascial plane, circumferential margin, ≥1 mm; distal margin, ≥1 mm]) and 30-day morbidity. End points were analyzed using surgeon quartiles defined by tool scores.

Results

The laparoscopic total mesorectal excision performance tool was produced and shown to be reliable and valid for the specialist level (intraclass correlation coefficient, 0.889; 95% CI, 0.832-0.926; P < .001). A substantial variation in tool scores was recorded (range, 25-48). Scores were associated with the number of intraoperative errors, plane of mesorectal dissection, and short-term patient morbidity, including the number and severity of complications. Upper quartile-scoring surgeons obtained excellent results compared with the lower quartile (mesorectal fascial plane: 93% vs 59%; number needed to treat [NNT], 2.9, P = .002; ALaCaRT end point success, 83% vs 58%; NNT, 4; P = .03; 30-day morbidity, 23% vs 50%; NNT, 3.7; P = .03).

Conclusions and Relevance

Intraoperative surgical skill can be objectively and reliably measured in complex cancer interventions. Substantial variation in technical performance among credentialed surgeons is seen and significantly associated with clinical and pathological outcomes.

Comparison of Performance Score for Female and Male Residents in General Surgery Doing Supervised Real-Life Laparoscopic Appendectomy: Is There a Norse Shield-Maiden Effect?

BACKGROUND

Gender bias may represent a threat to resident assessment during surgical training, and there have been concerns that women might be disadvantaged. There is a lack of studies investigating gender differences in 'entry-level' real-life procedures, such as laparoscopic appendectomy. We aimed to explore potential gender disparities in self-evaluation and faculty evaluation of a basic surgical procedure performed by junior surgical residents in general surgery.

METHODS

A structured training program in laparoscopic appendectomy was implemented before undertaking evaluation of real-life consecutive laparoscopic appendectomies by junior residents in general surgery. Resident and faculty gender-pairs were assessed. Intraclass correlation coefficient (ICC) was calculated using a single-rater, consistency, 2-way mixed-effects model.

RESULTS

A total of 165 paired sessions were completed to evaluate resident-faculty scores for the procedure. Overall, 19 residents participated (43% women) and 26 faculty (42% women) were involved. The overall correlation between faculty and residents was good (ICC > 0.8). The female-female pairs scored higher for most steps, achieving excellent (ICC ≥ 0.9) for several steps and for overall performance. Female residents were more likely to give a higher self-evaluated score on own performance particularly if evaluated by a female faculty. Also, female trainees had highest correlation-score with male faculty.

CONCLUSIONS

This study found higher performance scores in female surgical residents evaluated during real-time laparoscopic appendectomy. No negative gender bias toward women was demonstrated. Better insight into the dynamics of gender-based interaction and dynamics in both training, feedback and influence on evaluation during training is needed when evaluating surgical training programs.

Evaluating Non-operative Robotic Skills in Colorectal Surgical Training

BACKGROUND

Robotic-assisted surgery has become a common platform for performing colorectal procedures. Educators must determine how best to teach and train residents to use the technology safely. There is a paucity of literature on how non-operative skills are being taught and integrated into colorectal training. Herein we describe the implementation and assessment of a pilot simulation-based Robotic Colorectal Surgery Non-Technical Skills curriculum.

MATERIALS AND METHODS

Since 2017 six colon and rectal surgery residents participated in two scenarios: pelvic bleeding and CO₂ embolism. The scenarios were administered in a simulated operating room twice during the academic year (fall and spring), and audio-video recorded. In addition to self-assessment, videos were evaluated by faculty utilizing the validated Interpersonal and Cognitive Assessment for Robotic Surgery system. To understand the role of scenario difficulty with respect to perceived cognitive workload and performance residents completed a NASA-Task Load Index assessment form.

RESULTS

Between the fall and spring sessions residents significantly improved in intraoperative leadership skills for both the CO₂ embolism and bleeding scenarios, and decision-making and situational awareness for the embolism case. Assessment between resident (self) and expert (faculty) did not correlate (P < 0.05) for either scenario during the fall session. A correlation for both scenarios was appreciated following the spring session revealing resident non-technical skills improved over time. Other than for physical demand, NASA-Task Load Index scores were similar for both scenarios.

CONCLUSIONS

We were able to successfully develop and implement a pilot Robotic Colorectal Surgery Non-Technical Skills curriculum in a risk-free simulated environment. Non-technical skill curriculums should be considered for both training and assessment in robotic surgery.

European Society of Coloproctology Colorectal Robotic Surgery Training for the Trainers Course - the first pilot experience

AIM

Currently, there is no established colorectal specific robotic surgery Train the Trainer (TTT) course. The aim was to develop and evaluate such a course which can then be further developed to be incorporated within the planned European Society of Coloproctology (ESCP)/European School of Coloproctology (ESC) robotic colorectal surgery training curriculum.

METHOD

After identifying the need for such a course within a training programme, the course was developed by a subgroup of the ESCP/ESC. A scoping literature review was performed and the content and materials for the course were developed by a team consisting of two gastroenterologists with a combined experience of 30 years of facilitating TTT courses, a robotic surgeon and proctor with laparoscopic TTT faculty experience and experienced robotic and laparoscopic colorectal trainers. The course was evaluated by asking delegates to complete pre- and post-course questionnaires.

RESULTS

There were eight delegates on the course from across Europe. Delegates increased their knowledge of each of the course learning objectives and identified learning points in order to change practice. The feedback from the delegates of the course was positive across several areas and all felt that they had achieved their own personal objectives in attending the course.

CONCLUSION

This pilot robotic colorectal TTT course has achieved its aim and demonstrated many positives. There is a need for such a course and the evaluation processes have provided opportunities for reflection, which will allow the development/tailoring of future robotic colorectal TTT courses to help develop robotic training further.

Laparoscopic Colorectal Surgery Outcomes Improved After National Training Program (LAPCO) for Specialists in England

OBJECTIVE

To examine the impact of The National Training Program for Lapco on the rate of laparoscopic surgery and clinical outcomes of cases performed by Lapco surgeons after completion of training.

SUMMARY OF BACKGROUND DATA

Lapco provided competency-based supervised clinical training for specialist colorectal surgeons in England.

METHODS

We compared the rate of laparoscopic surgery, mortality, and morbidity for colorectal cancer resections by Lapco delegates and non-Lapco surgeons in 3-year periods preceding and following Lapco using difference in differences analysis. The changes in the rate of post-Lapco laparoscopic surgery with the Lapco sign-off competency assessment and in-training global assessment scores were examined using risk-adjusted cumulative sum to determine their predictive clinical validity with predefined competent scores of 3 and 5 respectively.

RESULTS

One hundred eight Lapco delegates performed 4586 elective colo-rectal resections pre-Lapco and 5115 post-Lapco while non-Lapco surgeons performed 72,930 matched cases. Lapco delegates had a 37.8% increase in laparoscopic surgery which was greater than non-Lapco surgeons by 20.9% [95% confidence interval (CI), 18.5-23.3, P < 0.001) with a relative decrease in 30-day mortality by -1.6% (95% CI, -3.4 to -0.2, P = 0.039) and 90-day mortality by -2.3% (95% CI, -4.3 to -0.4, P = 0.018). The change point of risk-adjusted cumulative sum was 3.12 for competency assessment tool and 4.74 for global assessment score whereas laparoscopic rate increased from 44% to 66% and 40% to 56%, respectively.

CONCLUSIONS

Lapco increased the rate of laparoscopic colorectal cancer surgery and reduced mortality and morbidity in England. In-training competency assessment tools predicted clinical performance after training.

Development and validation of a recommended checklist for assessment of surgical videos quality: the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool

INTRODUCTION

There has been a constant increase in the number of published surgical videos with preference for open-access sources, but the proportion of videos undergoing peer-review prior to publication has markedly decreased, raising questions over quality of the educational content presented. The aim of this study was the development and validation of a standard framework for the appraisal of surgical videos submitted for presentation and publication, the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool.

METHODS

An international committee identified items for inclusion in the LAP-VEGaS video assessment tool and finalised the marking score utilising Delphi methodology. The tool was finally validated by anonymous evaluation of selected videos by a group of validators not involved in the tool development.

RESULTS

9 items were included in the LAP-VEGaS video assessment tool, with every item scoring from 0 (item not presented in the video) to 2 (item extensively presented in the video), with a total marking score ranging from 0 to 18. The LAP-VEGaS video assessment tool resulted highly accurate in identifying and selecting videos for acceptance for conference presentation and publication, with high level of internal consistency and generalisability.

CONCLUSIONS

We propose that peer review in adherence to the LAP-VEGaS video assessment tool could enhance the overall quality of published video outputs.

Self-Assessment of Surgical Skills: A Systematic Review

OBJECTIVE

Self-assessment is fundamental in surgical training to enhance learning in the absence of trainer feedback. The primary objective of this review was to assess the factors that influence accuracy of self-assessment at technical skills across all surgical specialties. The secondary objective was to assess whether there are any innate factors or attributes to predict those that will carry out effective self-assessment.

DESIGN

A systematic review was carried out in accordance with PRISMA guidelines. A search strategy encompassing MEDLINE, EMBASE, ERIC, WHO, and the Cochrane database was conducted to identify studies investigating self-assessment at any surgical task. Quality was assessed using the Newcastle-Ottawa scale. A summary table was created to describe specialty, participants, task, setting, assessment tool, and correlation coefficient between self and expert assessment. The review protocol was registered in PROSPERO.

RESULTS

Of 24,638 citations, 40 met inclusion criteria. In total 1753 participants performed 68 procedures. Twenty-six studies investigated skills in general surgery with the remaining 14 in various other surgical specialties. Accuracy of self-assessment is superior in those with greater experience and age, and with use of retrospective video playback. Accuracy tends to be reflected by overestimation of performance. Stressful environments reduce accuracy. There is limited evidence in the literature regarding predicting traits for those who will carry out accurate self-assessment.

CONCLUSIONS

The ability to perform accurate self-assessment is an important skill in surgical training, with accuracy being influenced by a multitude of factors. The use of self-assessment from retrospective video playback may be of benefit in surgical training curricula to enhance learning of technical skills. Further studies are required to define predictors of good self-assessment, which will strengthen recruitment and mentoring to assist trainee learning.

Factors Predicting Operative Difficulty of Laparoscopic Total Mesorectal Excision

BACKGROUND

Laparoscopic total mesorectal excision is a challenging procedure requiring high-quality surgery for optimal outcomes. Patient, tumor, and pelvic factors are believed to determine difficulty, but previous studies were limited to postoperative data.

OBJECTIVE

This study aimed to report factors predicting laparoscopic total mesorectal excision performance by using objective intraoperative assessment.

DESIGN

Data from a multicenter laparoscopic total mesorectal excision randomized trial (ISRCTN59485808) were reviewed.

SETTING

This study was conducted at 4 centers in the United Kingdom.

PATIENTS AND INTERVENTION

Seventy-one patients underwent elective laparoscopic total mesorectal excision for rectal adenocarcinoma with curative intent: 53% were men, mean age was 69 years, body mass index was 27.7, tumor height was 8.5 cm, 24% underwent neoadjuvant therapy, and 25% had previous surgery.

MAIN OUTCOME MEASURES

Surgical performance was assessed through the identification of intraoperative adverse events by using observational clinical human reliability analysis. Univariate analysis and multivariate binomial regression were performed to establish factors predicting the number of intraoperative errors, surgeon-reported case difficulty, and short-term clinical and histopathological outcomes.

RESULTS

A total of 1331 intraoperative errors were identified from 365 hours of surgery (median, 18 per case; interquartile range, 16-22; and range, 9-49). No patient, tumor, or bony pelvimetry measurement correlated with total or pelvic error count, surgeon-reported case difficulty, cognitive load, operative data, specimen quality, number or severity of 30-day morbidity events and length of stay (all r not exceeding ±0.26, p > 0.05). Mesorectal area was associated with major intraoperative adverse events (OR, 1.09; 95%CI, 1.01-1.16; p = 0.015) and postoperative morbidity (OR, 1.1; 95% CI, 1.01-1.2; p = 0.033). Obese men were subjectively reported as harder cases (24 vs 36 mm, p = 0.042), but no detrimental effects on performance or outcomes were seen.

LIMITATIONS

Our sample size is modest, risking type II errors and overfitting of the statistical models.

CONCLUSION

Patient, tumor, and bony pelvic anatomical characteristics are not seen to influence laparoscopic total mesorectal excision operative difficulty. Mesorectal area is identified as a risk factor for intraoperative and postoperative morbidity. See Video Abstract at http://links.lww.com/DCR/B35. FACTORES QUE PREDICEN LA DIFICULTAD OPERATIVA DE LA ESCISIÓN MESORRECTAL TOTAL LAPAROSCÓPICA: La escisión mesorrectal total laparoscópica es un procedimiento desafiante. Para obtener resultados óptimos, se requiere cirugía de alta calidad. Se cree que, factores como el paciente, el tumor y la pelvis, determinan la dificultad, pero estudios previos solamente se han limitado a datos postoperatorios.Informar de los factores que predicen el resultado de la escisión mesorrectal total laparoscópica, mediante una evaluación intraoperatoria objetiva.Datos de un ensayo multicéntrico y randomizado de escisión mesorrectal total laparoscópica (ISRCTN59485808).Cuatro centros del Reino Unido.Un total de 71 pacientes fueron sometidos a escisión mesorrectal total laparoscópica electiva, para adenocarcinoma rectal con intención curativa. 53% hombres, edad media, índice de masa corporal y altura del tumor 69, 27.7 y 8.5 cm respectivamente, 24% terapia neoadyuvante y 25% cirugía previa.Rendimiento quirúrgico evaluado mediante la identificación de eventos intraoperatorios adversos, mediante el análisis clínico observacional de confiabilidad humana. Se realizaron análisis univariado y la regresión binomial multivariada para establecer factores que predicen el número de errores intraoperatorios, reportes del cirujano sobre la dificultad del caso y los resultados clínicos e histopatológicos a corto plazo.Se identificaron un total de 1,331 errores intraoperatorios en 365 horas de cirugía (media de 18 por caso, IQR 16-22, rango 9-49). Ningún paciente, tumor o medición de pelvimetría pélvica, se correlacionó con la cuenta de errores pélvicos o totales, reporte del cirujano sobre dificultad del caso, carga cognitiva, datos operativos, calidad de la muestra, número o gravedad de eventos de morbilidad de 30 días y duración de la estadía (todos r <± 0.26, p > 0.05). El área mesorrectal se asoció con eventos adversos intraoperatorios importantes (OR, 1.09; IC 95%, 1.01-1.16; p = 0.015) y morbilidad postoperatoria (OR, 1.1; IC 95%, 1.01-1.2; p = 0.033). Como información subjetiva, hombres obesos fueron casos más difíciles (24 mm frente a 36 mm, p = 0.042) pero no se observaron efectos perjudiciales sobre el rendimiento o los resultados.Nuestro tamaño de muestra es un modesto riesgo de errores de tipo II y el sobreajuste de los modelos estadísticos.No se observa que las características anatómicas del paciente, tumor y pelvis ósea influyan en la dificultad operatoria de la escisión mesorrectal laparoscópica total. El área mesorrectal se identifica como un factor de riesgo para la morbilidad intraoperatoria y postoperatoria. Vea el resumen del video en http://links.lww.com/DCR/B35.

Video-based coaching in surgical education: a systematic review and meta-analysis

BACKGROUND

In the era of competency-based surgical education, VBC has gained increased attention and may enhance the efficacy of surgical education. The objective of this systematic review was to summarize the existing evidence of video-based coaching (VBC) and compare VBC to traditional master-apprentice-based surgical education.

METHODS

We performed a systematic review and meta-analysis of randomized controlled trials (RCT) assessing VBC according to the PRISMA and Cochrane guidelines. The MEDLINE, EMBASE, and COCHRANE and Researchgate databases were searched for eligible manuscripts. Standard mean difference (SMD) of performance scoring scales was used to assess the effect of VBC versus traditional training without VBC (control).

RESULTS

Of 627 studies identified, 24 RCTs were eligible and evaluated. The studies included 778 surgical trainees (n = 386 VBC vs. n = 392 control). 13 performance scoring scales were used to assess technical competence; OSATS-GRS was the most common (n = 15). VBC was provided preoperative (n = 11), intraoperative (n = 1), postoperative (n = 10), and perioperative (n = 2). The majority of studies were unstructured, where identified coaching frameworks were PRACTICE (n = 1), GROW (n = 2) and Wisconsin Coaching Framework (n = 1). There was an effect on performance scoring scales in favor of VBC coaching (SMD 0.87, p < 0.001). In subgroup analyses, the residents had a larger relative effect (SMD 1.13; 0.61-1.65, p < 0.001) of VBC compared to medical students (SMD 0.43, 0.06-0.81, p < 0.001). The greatest source of potential bias was absence of blinding of the participants and personnel (n = 20).

CONCLUSION

Video-based coaching increases technical performance of medical students and surgical residents. There exist significant study and intervention heterogeneity that warrants further exploration, showing the need to structure and standardize video-based coaching tools.

Expert consensus on a train-the-trainer curriculum for robotic colorectal surgery

AIM

Robotic techniques are being increasingly used in colorectal surgery. There is, however, a lack of training opportunities and structured training programmes. Robotic surgery has specific problems and challenges for trainers and trainees. Ergonomics, specific skills and user-machine interfaces are different from those in traditional laparoscopic surgery. The aim of this study was to establish expert consensus on the requirements for a robotic train-the-trainer curriculum amongst robotic surgeons and trainers.

METHOD

This is a modified Delphi-type study involving 14 experts in robotic surgery teaching. A reiterating 19-item questionnaire was sent out to the same group and agreement levels analysed. A consensus of 0.8 or higher was considered to be high-level agreement.

RESULTS

Response rates were 93-100% and most items reached high levels of agreement within three rounds. Specific requirements for a robotic faculty development curriculum included maximizing dual-console teaching, theatre team training, nontechnical skills training, patient safety, user-machine interface training and telementoring.

CONCLUSION

A clear need for the development of a train-the-trainer curriculum has been identified. Further research is needed to assess feasibility, effectiveness and clinical impact of a robotic train-the-trainer curriculum.

Evaluation tool for a gastroenterostomy simulated training1

PURPOSE

To create a checklist to evaluate the performance and systematize the gastroenterostomy simulated training.

METHODS

Experimental longitudinal study of a quantitative character. The sample consisted of twelve general surgery residents. The training was divided into 5 sessions and consisted of participation in 20 gastroenterostomys in synthetic organs. The training was accompanied by an experienced surgeon who was responsible for the feedback and the anastomoses evaluation. The anastomoses evaluated were the first, fourth, sixth, eighth and tenth. A 10 item checklist and the time to evaluate performance were used.

RESULTS

Residents showed a reduction in operative time and evolution in the surgical technique statistically significant (p<0.01). The correlation index of 0.545 and 0,295 showed a high linear correlation between time variables and Checklist. The average Checklist score went from 6.8 to 9 points.

CONCLUSION

The proposed checklist can be used to evaluate the performance and systematization of a simulated training aimed at configuring a gastroenterostomy.

Assessment of surgical performance of laparoscopic benign hiatal surgery: a systematic review

BACKGROUND

Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument.

METHODS

A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences).

RESULTS

There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF.

CONCLUSION

There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.

Monitoring excess unplanned return to theatre following colorectal cancer surgery

BACKGROUND

To develop a risk-adjustment model for unplanned return to theatre (URTT) outcomes following colorectal surgeries in Australia and New Zealand hospitals and apply top-down and bottom-up statistical process control methods for fair comparison of hospitals and surgeons' URTT rates.

METHODS

We analysed URTT outcomes from hospitals contributing data to the Bi-National Colorectal Cancer Audit clinical registry between 2007 and 2016. Preoperative and intraoperative covariates were considered for risk adjustment. A risk-adjusted rate funnel plot was prepared for between-hospital comparisons and identification of outlying hospitals with unusually high rates of URTT. Cumulative observed-minus-expected charts with cumulative sum signals were then presented for surgeons within an outlying hospital.

RESULTS

The study included 15 134 patients and 166 surgeons across 70 hospitals. The weighted average URTT rate was 5.2%. The risk-adjustment model identified 12 preoperative and intraoperative variables that significantly raise the risk of URTT: male sex, American Society of Anesthesiologists score, emergency admissions, conversion entry, left hemicolectomy, total colectomy, proctocolectomy, lower anterior resection, ultra-low anterior resection, abdominoperineal resection, organ resection and excess lymph nodes harvested. Right hemicolectomy significantly reduced risk of URTT. URTT rates were not found to significantly vary across seniority of operator; however, comparisons were limited by lack of data on junior operators. The funnel plot identified five hospitals as 'possible outliers' and hospital T was identified as a 'definite outlier'. The cumulative observed-minus-expected charts with cumulative sum signals showed that within hospital T, one surgeon among three had a particularly bad run of URTTs.

CONCLUSION

Feedback from aggregated URTT outcomes using a risk-adjusted rate funnel plot is enhanced when follow-up examination of outlying hospitals is conducted with concurrent application of cumulative observed-minus-expected charts with cumulative sum signals.

EAES classification of intraoperative adverse events in laparoscopic surgery

BACKGROUND

Surgical outcomes are traditionally evaluated by post-operative data such as histopathology and morbidity. Although these outcomes are reported using accepted systems, their ability to influence operative performance is limited by their retrospective application. Interest in direct measurement of intraoperative events is growing but no available systems applicable to routine practice exist. We aimed to develop a structured, practical method to report intraoperative adverse events enacted during minimal access surgical procedures.

METHODS

A structured mixed methodology approach was adopted. Current intraoperative adverse event reporting practices and desirable system characteristics were sought through a survey of the EAES executive. The observational clinical human reliability analysis method was applied to a series of laparoscopic total mesorectal excision (TME) case videos to identify intraoperative adverse events. In keeping with survey results, observed events were further categorised into non-consequential and consequential, which were further subdivided into four levels based upon the principle of therapy required to correct the event. A second survey phase explored usability, acceptability, face and content validity of the novel classification.

RESULTS

217 h of TME surgery were analysed to develop and continually refine the five-point hierarchical structure. 34 EAES expert surgeons (69%) responded. The lack of an accepted system was the main barrier to routine reporting. Simplicity, reproducibility and clinical utility were identified as essential requirements. The observed distribution of intraoperative adverse events was 60.1% grade I (non-consequential), 37.1% grade II (minor corrective action), 2.4% grade III (major correction or change in post-operative care) and 0.1% grade IV (life threatening). 84% agreed with the proposed classification (Likert scale 4.04) and 92% felt it was applicable to their practice and incorporated all desirable characteristics.

CONCLUSION

A clinically applicable intraoperative adverse event classification, which is acceptable to expert surgeons, is reported and complements the objective assessment of minimal access surgical performance.

Developing an objective assessment of surgical performance from operating room video and surgical imagery

An unbiased, repeatable process for assessing operating room performance is an important step toward quantifying the relationship between surgical training and performance. Hip fracture surgeries offer a promising first target in orthopedic trauma because they are common and they offer quantitative performance metrics that can be assessed from video recordings and intraoperative fluoroscopic images. Hip fracture repair surgeries were recorded using a head-mounted point-of-view camera. Intraoperative fluoroscopic images were also saved. The following performance metrics were analyzed: duration of wire navigation, number of fluoroscopic images collected, degree of intervention by the surgeon's supervisor, and the tip-apex distance (TAD). Two orthopedic traumatologists graded surgical performance in each video independently using an Objective Structured Assessment of Technical Skill (OSATS). Wire navigation duration correlated with weeks into residency and prior cases logged. TAD correlated with cases logged. There was no significant correlation between the OSATS total score and experience metrics. Total OSATS score correlated with duration and number of fluoroscopic images. Our results indicate that two metrics of hip fracture wire navigation performance, duration and TAD, significantly differentiate surgical experience. The methods presented have the potential to provide truly objective assessment of resident technical performance in the OR.

Mentored Trainees have Similar Short-Term Outcomes to a Consultant Trainer Following Laparoscopic Colorectal Resection

BACKGROUND

Laparoscopic colorectal surgery has a long learning curve. Using a modular-based training programme may shorten this. Concerns with laparoscopic surgery have been oncological compromise and poor surgical outcomes when training more junior surgeons. This study aimed to compare operative and oncological outcomes between trainees undergoing a mentored training programme and a consultant trainer.

METHODS

A prospective study of all elective laparoscopic colorectal resections was undertaken in a single institution. Operative and oncological outcomes were recorded. All trainees were mentored by a National Laparoscopic Trainer (Lapco), and results between trainer and trainees compared.

RESULTS

Three hundred cases were included, with 198 (66%) performed for cancer. The trainer undertook 199 (66%) of operations, whilst trainees performed 101 (34%). Anterior resection was the commonest operation (n = 124, 41%). There were no differences between trainer and trainees for the majority of surgical outcomes, including blood loss (p = 0.598), conversion to open (p = 0.113), anastomotic leak (p = 0.263), readmission (p = 1.000) and death rates (p = 0.549). Only length of stay (p = 0.034), stoma formation (p < 0.01) and operative duration (p = 0.007) were higher in the trainer cohort, reflecting the more complex cases undertaken. Overall, there were no significant differences in both short- and longer-term oncology outcomes according to the grade of operating surgeon, including lymph nodes in specimen, circumferential resection margin and 1- and 2-year radiological recurrence.

CONCLUSION

When a modular-based training system was combined with case selection, both clinical and histopathological outcomes following resectional laparoscopic colorectal surgery were similar between trainees and trainer. This should encourage the use of more training opportunities in laparoscopic colorectal surgery.

The American Society of Colon and Rectal Surgeons Assessment Tool for Performance of Laparoscopic Colectomy

BACKGROUND

The lack of consensus for performance assessment of laparoscopic colorectal resection is a major impediment to quality improvement.

OBJECTIVE

The purpose of this study was to develop and assess the validity of an evaluation tool for laparoscopic colectomy that is feasible for wide implementation.

DESIGN

During the pilot phase, a small group of experts modified previous assessment tools by watching videos for laparoscopic right colectomy with the following categories of experience: novice (less than 20 cases), intermediate (50-100 cases), and expert (more than 500 cases). After achieving sufficient reliability (κ > 0.8), a user-friendly tool was validated among a large group of blinded, trained experts.

SETTING

The study was conducted through the American Society of Colon and Rectal Surgeons Operative Competency Evaluation Committee.

PATIENTS

Raters were from the Operative Competency Evaluation Committee of the American Society of Colon and Rectal Surgeons.

MAIN OUTCOME MEASURES

Assessment tool reliability and internal consistency were measured.

RESULTS

From October 2014 through February 2015, 4 groups of 5 raters blinded to surgeon skill level evaluated 6 different laparoscopic right colectomy videos (novice = 2, intermediate = 2, expert = 2). The overall Cronbach α was 0.98 (>0.9 = excellent internal consistency). The intraclass correlation for the overall assessment was 0.93 (range, 0.77-0.93) and was >0.74 (excellent) for each step. The average scores (scale, 1-5) for experts were significantly better than those in the intermediate category, with a mean (SD) of 4.51 (0.56) versus 2.94 (0.56; p = 0.003). Videos in the intermediate group scored more favorably than beginner videos for each individual step and overall performance (mean (SD) = 3.00 (0.32) vs 1.78 (0.42); p = 0.006).

LIMITATIONS

The study was limited by rater bias to technique and style.

CONCLUSIONS

The unique and robust methodology in this trial produced an assessment tool that was feasible for raters to use when assessing videotaped laparoscopic right hemicolectomies. The potential applications for this new tool are widespread, including both training and evaluation of competence at the attending level. See Video Abstract at http://links.lww.com/DCR/A369, http://links.lww.com/DCR/A370, http://links.lww.com/DCR/A371.

Objective assessment of minimally invasive total mesorectal excision performance: a systematic review

INTRODUCTION

Laparoscopy is widely used in colorectal practice, but recent trial results have questioned its use in rectal cancer resections. Patient outcomes are directly linked to the quality of total mesorectal excision (TME) specimen. Objective assessment of intraoperative performance could help ensure competence and delivery of optimal outcomes. Objective tools may also contribute to TME intervention trials, but their nature, structure and utilisation is unknown.

AIM

To systemically review the available literature to report on the available tools for the objective assessment of minimally invasive TME operative performance and their use within multicentre laparoscopic TME randomised controlled trials.

METHODS

A systematic search of the PubMed and Cochrane databases was performed to identify tools used in the objective intraoperative assessment of minimally invasive TME performance in accordance with the PRISMA guidelines, independently by two authors. The identified tools were then evaluated within reported TME RCTs.

RESULTS

A total of 8642 abstracts were screened of which 12 papers met the inclusion criteria; ten prospective observational studies, one randomised trial and one educational consensus. Eight assessment methods were described, which include formative and summative tools. The tools were mostly adaptations of colonic surgery tools based on either operative video review or post-operative trainer rating. All studies reported objective assessment of intraoperative performance was feasible, but only 126 (7%) of the 1762 included laparoscopic cases were TME. No multicentre laparoscopic TME trial reported using any objective surgical performance assessment tool.

CONCLUSION

Objective intraoperative laparoscopic TME performance assessment is feasible, but most of the current tools are adaptation of colonic surgery. There is a need to develop dedicated assessment tools for minimal access rectal surgery. No multicentre minimally invasive TME RCT reported using any objective assessment tool.

The impact of the introduction of formalised polypectomy assessment on training in the UK

OBJECTIVE

The aim was to describe the impact on polypectomy experience by the mandatory introduction of the Directly Observed Polypectomy Skills tool (DOPyS) and electronic portfolio as part of the formal colonoscopy certification process.

DESIGN

Applications for colonoscopy certification in the UK in the year prior to the introduction of DOPyS were analysed retrospectively and compared with data collected prospectively for those in the following year.

SETTING

UK National Health Service.

PATIENTS

None.

INTERVENTIONS

None.

MAIN OUTCOME MEASURES

The outcomes studied included whether evidence of exposure to polypectomy, endoscopic mucosal resection (EMR) and colonoscopy changed over the 2-year period. The nature of the polyps removed by trainees was also studied.

RESULTS

Thirty two per cent of candidates in the first year had evidence of any observed polypectomy with 7% of candidates referring to training in EMR. The median number of formative colonoscopy assessments was 3 (range 0-16). All of these candidates in the second year had evidence of polypectomy assessment, with a median number of DOPyS of 7 (range 3-27). Eighty nine per cent of applicants had evidence of assessed EMR. The median number of formative colonoscopy assessments in this cohort was 32 (range 9-199). There was a significant increase in the number of logged polypectomy assessments (p<0.001), experience of EMR (p<0.001) and formative colonoscopy assessments (p<0.001). There was no significant difference in the total number of colonoscopy procedures performed.

CONCLUSIONS

Structured polypectomy assessment improves trainees' documented exposure to therapeutic endoscopy as well as providing formal evidence of skills acquisition. As polypectomy plays an increasing role globally in colorectal cancer prevention, the DOPyS provides an effective means of assessing and certifying polypectomy.

[Risk awareness and training for prevention of complications in minimally invasive surgery]

BACKGROUND

Minimally invasive surgery (MIC) requires surgeons to have a different set of skills and capabilities from that of open surgery. The indirect camera view, lack of a three-dimensional view, restricted haptic feedback with lack of tissue feeling and difficult instrument coordination with fulcrum and pivoting effects result in an additional learning curve compared to open surgery. The prolonged learning curve leads to a higher risk of complications and special awareness of these risks is therefore mandatory. Training of special laparoscopic skills outside the operating room is needed to optimize patient outcome and to minimize the ocurrence of complications related to the learning curve.

RESULTS AND DISCUSSION

Training modalities for laparoscopic surgery include simple box trainers, computer simulation with virtual reality, the use of artificial and cadaver organs, as well as live animal models and cadaver training. These training modalities have been proven in studies to have a beneficial effect on the learning curve for acquisition of laparoscopic skills and for improving operative performance as well as avoidance of complications. Laparoscopic training is currently gaining a more and more important role for official education and accreditation purposes. In some countries the participation in laparoscopic training courses has become mandatory prior to participation in laparoscopic operations. Future research will include the optimization of multimodal training curricula, the development of individualized training approaches that allow both trainee and patient-specific preparation, as well as the use of novel devices to facilitate the collection and transfer of expertise between the generations and schools of surgeons.

Assessing Wire Navigation Performance in the Operating Room

OBJECTIVE

There are no widely accepted, objective, and reliable tools for measuring surgical skill in the operating room (OR). Ubiquitous video and imaging technology provide opportunities to develop metrics that meet this need. Hip fracture surgery is a promising area in which to develop these measures because hip fractures are common, the surgery is used as a milestone for residents, and it demands technical skill. The study objective is to develop meaningful, objective measures of wire navigation performance in the OR.

DESIGN

Resident surgeons wore a head-mounted video camera while performing surgical open reduction and internal fixation using a dynamic hip screw. Data collected from video included: duration of wire navigation, number of fluoroscopic images, and the degree of intervention by the surgeon׳s supervisor. To determine reliability of these measurements, 4 independent raters performed them for 2 cases. Raters independently measured the tip-apex distance (TAD), which reflects the accuracy of the surgical placement of the wire, on all the 7 cases.

SETTING

University of Iowa Hospitals and Clinics in Iowa City, IA-a public tertiary academic center.

PARTICIPANTS

In total 7 surgeries were performed by 7 different orthopedic residents. All 10 raters were biomedical engineering graduate students.

RESULTS

The standard deviations for anteroposterior, lateral, and combined TAD measurements of the 10 raters were 2.7, 1.9, and 3.7mm, respectively, and interrater reliability produced a Cronbach α of 0.97. The interrater reliability analysis for all 9 video-based measures produced a Cronbach α of 0.99.

CONCLUSIONS

Several video-based metrics were consistent across the 4 video reviewers and are likely to be useful for performance assessment. The TAD measurement was less reliable than previous reports have suggested, but remains a valuable metric of performance. Nonexperts can reliably measure these values and they offer an objective assessment of OR performance.

Failing to Prepare Is Preparing to Fail: A Single-Blinded, Randomized Controlled Trial to Determine the Impact of a Preoperative Instructional Video on the Ability of Residents to Perform Laparoscopic Right Colectomy

BACKGROUND

Laparoscopic colorectal resection is an index case for advanced skills training, yet many residents struggle to reach proficiency by graduation. Current methods to reduce the learning curve for residents remain expensive, time consuming, and poorly validated.

OBJECTIVE

The purpose of this study was to assess the impact of the addition of a preprocedural instructional video to improve the ability of a general surgery resident to perform laparoscopic right colectomy when compared with standard preparation.

DESIGN

This was a single-blinded, randomized control study.

SETTINGS

Four university-affiliated teaching hospitals were included in the study.

PARTICIPANTS

General surgery residents in postgraduation years 2 through 5 participated.

INTERVENTION

Residents were randomly assigned to preparation with a narrated instructional video versus standard preparation.

MAIN OUTCOME MEASURES

Resident performance, scored by a previously validated global assessment scale, was measured.

RESULTS

Fifty-four residents were included. Half (n = 27) were randomly assigned to view the training video and half (n = 27) to standard preparation. There were no differences between groups in terms of training level or previous operative experience or in patient demographics (all p > 0.05). Groups were similar in the percentage of the case completed by residents (p = 0.39) and operative time (p = 0.74). Residents in the video group scored significantly higher in total score (mean: 46.8 vs 42.3; p = 0.002), as well as subsections directly measuring laparoscopic skill (vascular control mean: 11.3 vs 9.7, p < 0.001; mobilization mean: 7.6 vs. 7.0, p = 0.03) and overall performance score (mean: 4.0 vs 3.1; p < 0.001). Statistical significance persisted across training levels.

LIMITATIONS

There is potential for Hawthorne effect, and the study is underpowered at the individual postgraduate year level.

CONCLUSIONS

The simple addition of a brief, narrated preprocedural video to general surgery resident case preparation significantly increased trainee ability to successfully perform a laparoscopic right colectomy. In an era of shortened hours and less exposure to cases, incorporating a brief but effective instructional video before surgery may improve the learning curve of trainees and ultimately improve safety.

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Background

There is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).

Methods

An independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.

Results

Independence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).

Discussion/conclusion

A procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

Outreach training model for accredited colorectal specialists in laparoscopic colorectal surgery: feasibility and evaluation of challenges

AIM

The aim of this study was to explore the feasibility and safety of an outreach model of laparoscopic colorectal training of accredited specialists in advanced laparoscopic techniques and to explore the challenges of this model from the perspective of a National Training Programme (NTP) trainer.

METHOD

Prospective data were collected for unselected laparoscopic colorectal training procedures performed by five laparoscopic colorectal NTP trainees supervised by a single NTP trainer with an outreach model between 2009 and 2012. The operative and postoperative outcomes were compared with standard laparoscopic colorectal training procedures performed by six senior colorectal trainees under the supervision of the same NTP trainer within the same study period. The primary outcome was 30-day mortality. The Mann-Whitney test was used to compare continuous variables and the Chi squared or Fisher's exact tests were applied for the analysis of categorical variables. The level of statistical significance was set at P < 0.05.

RESULTS

During the study period 179 elective laparoscopic colorectal procedures were performed. This included 54 cases performed by NTP trainees and 125 cases performed by the supervised trainees. There were no significant differences in age, gender, body mass index, American Society of Anesthesiologists grade, pathology and procedure type between both groups. Seventy-eight per cent of the patients operated on by the NTP trainees had had no previous abdominal surgery, compared with 50% in the supervised trainees' group (P = 0.0005). There were no significant differences in 30-day mortality or the operative and postoperative outcome between both groups. There were, however, difficulties in training an already established consultant in his or her own hospital and these were overcome by certain adjustments to the programme.

CONCLUSION

Outreach laparoscopic training of colorectal surgeons is a feasible and safe model of training accredited specialists and does not compromise patient care. The challenges encountered can be overcome with optimum training and preparation.

What errors make a laparoscopic cancer surgery unsafe? An ad hoc analysis of competency assessment in the National Training Programme for laparoscopic colorectal surgery in England

BACKGROUND

The National Training Programme for laparoscopic colorectal surgery in England was implemented to ensure training was supervised, structured, safe and effective. Delegates were required to pass a competency assessment (sign-off) before undertaking independent practice. This study described the types of errors identified and associated these errors with competency to progress to independent laparoscopic colorectal practice.

METHODS

All sign-off submissions from the start of the process in January 2008 until July 2013 were included. Content analysis was used to categorise errors. Bayes factor (BF) was used to measure the impact of individual error on assessment outcome. A smaller BF indicates that an error has stronger associations with unsuccessful assessments. Bayesian network was employed to graphically represent the reasoning process whereby the chance of successful assessment diminished with the identification of each error. Quality of the error feedback was measured by the area under the ROC curve which linked the predictions from the Bayesian model to the expert verdict.

RESULTS

Among 370 assessments analysed, 240 passed and 130 failed. On average, 2.5 different types of error were identified in each assessment. Cases that were more likely to fail had three or more different types of error (χ(2) = 72, p < 0.0001) and demonstrated poorer technical skills (CAT score <2.7, χ(2) = 164, p < 0.0001). Case complexity or right- versus left-sided resection did not have a significant impact. Errors associated with dissection (BF = 0.18), anastomosis (BF = 0.23) and oncological quality (BF = 0.19) were critical determinants of surgical competence, each reducing the odds of pass by at least fourfold. The area under the ROC curve was 0.84.

CONCLUSIONS

Errors associated with dissection, anastomosis and oncological quality were critical determinants of surgical competency. The detailed error analysis reported in this study can guide the design of future surgical education and clinical training programmes.