Feasibility of self-appraisal in assessing operative performance in advanced laparoscopic colorectal surgery

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

Assessment methods in laparoscopic colorectal surgery: a systematic review of available instruments

BACKGROUND

Laparoscopic surgery has become the golden standard for many procedures, requiring new skills and training methods. The aim of this review is to appraise literature on assessment methods for laparoscopic colorectal procedures and quantify these methods for implementation in surgical training.

MATERIALS AND METHODS

PubMed, Embase and Cochrane Central Register of Controlled Trials databases were searched in October 2022 for studies reporting learning and assessment methods for laparoscopic colorectal surgery. Quality was scored using the Downs and Black checklist. Included articles were categorized in procedure-based assessment methods and non-procedure-based assessment methods. A second distinction was made between capability for formative and/or summative assessment.

RESULTS

In this systematic review, nineteen studies were included. These studies showed large heterogeneity despite categorization. Median quality score was 15 (range 0-26). Fourteen studies were categorized as procedure-based assessment methods (PBA), and five as non-procedure-based assessment methods. Three studies were applicable for summative assessment.

CONCLUSIONS

The results show a considerable diversity in assessment methods with varying quality and suitability. To prevent a sprawl of assessment methods, we argue for selection and development of available high-quality assessment methods. A procedure-based structure combined with an objective assessment scale and possibility for summative assessment should be cornerstones.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review

Background

In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature.

Methods

A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools.

Results

Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings.

Conclusion

The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-021-08914-z.

Objective assessment of minimally invasive total mesorectal excision performance: a systematic review

INTRODUCTION

Laparoscopy is widely used in colorectal practice, but recent trial results have questioned its use in rectal cancer resections. Patient outcomes are directly linked to the quality of total mesorectal excision (TME) specimen. Objective assessment of intraoperative performance could help ensure competence and delivery of optimal outcomes. Objective tools may also contribute to TME intervention trials, but their nature, structure and utilisation is unknown.

AIM

To systemically review the available literature to report on the available tools for the objective assessment of minimally invasive TME operative performance and their use within multicentre laparoscopic TME randomised controlled trials.

METHODS

A systematic search of the PubMed and Cochrane databases was performed to identify tools used in the objective intraoperative assessment of minimally invasive TME performance in accordance with the PRISMA guidelines, independently by two authors. The identified tools were then evaluated within reported TME RCTs.

RESULTS

A total of 8642 abstracts were screened of which 12 papers met the inclusion criteria; ten prospective observational studies, one randomised trial and one educational consensus. Eight assessment methods were described, which include formative and summative tools. The tools were mostly adaptations of colonic surgery tools based on either operative video review or post-operative trainer rating. All studies reported objective assessment of intraoperative performance was feasible, but only 126 (7%) of the 1762 included laparoscopic cases were TME. No multicentre laparoscopic TME trial reported using any objective surgical performance assessment tool.

CONCLUSION

Objective intraoperative laparoscopic TME performance assessment is feasible, but most of the current tools are adaptation of colonic surgery. There is a need to develop dedicated assessment tools for minimal access rectal surgery. No multicentre minimally invasive TME RCT reported using any objective assessment tool.

Utilizing a novel tool for the comprehensive assessment of resident operative performance

PURPOSE

A mechanism for more effective and comprehensive assessment of surgical residents' performance in the operating room (OR) is needed, especially in light of the new requirements issued by the American Board of Surgery. Furthermore, there is an increased awareness that assessments need to be more meaningful by including not only procedure-specific and general technical skills, but also nontechnical skills (NOTECHS), such as teamwork and communication skills. Our aims were to develop a methodology and create a tool that comprehensively assesses residents' operative performance.

METHODS

A procedure-specific technical skill assessment for laparoscopic colon resections was created through use of task analysis. Components of previously validated tools were added to broaden the assessment to include general technical skills and NOTECHS. Our instrument was then piloted in the OR to measure face and content validity through an iterative process with faculty evaluators. Once the tool was finalized, postgraduate 3 (PG3) and PG5 residents on a 2-month long rotation were assessed by 1 of 4 colorectal surgeons immediately after completing a case together. Construct validity was measured by evaluating the difference in scores between PG3 and PG5 residents' performance as well as the change in scores over the course of the rotation.

RESULTS

Sixty-three assessments were performed. All evaluations were completed within 48 hours of the operation. There was a statistically significant difference between the PG3 and PG5 scores on procedure-specific performance, general technical skills, NOTECHS, and overall performance. Over the course of the rotation, a statistically significant improvement was found in residents' scores on the procedure-specific portion of the assessment but not on the general surgical skills or NOTECHS.

CONCLUSION

This is a feasible, valid, and reliable assessment tool for the comprehensive evaluation of resident performance in the OR. We plan to use this tool to assess resident operative skill development and to improve direct resident feedback.

Self versus external assessment for technical tasks in surgery: a narrative review

BACKGROUND

Self-assessment is an intricate component of continuing professional development and lifelong learning for health professionals. The agreement between self and external assessment for cognitive tasks in health professionals is reported to be poor; however, this topic has not been reviewed for technical tasks in surgery.

OBJECTIVE

To compare self and external assessment for technical tasks in surgery.

METHODS

MEDLINE, ERIC, and Google Scholar databases were searched for data from January 1960 to November 2011. Inclusion criteria were restricted to articles published in English in peer-reviewed journals, which reported on a comparison between self and external assessment for a technical task in a surgical specialty and involved medical students, surgical residents, surgical fellows, or practicing surgeons. Abstracts of identified articles were reviewed and pertinent full-text versions were retrieved. Manual searching of bibliographies for additional studies was performed. Data were extracted in a systematic manner.

RESULTS

From a total of 49 citations, 17 studies (35%) were selected for review. Eight of the 17 studies (47%) reported no agreement, whereas 9 studies (53%) reported an agreement between self and external assessment for technical tasks in surgery. Four studies (24%) reported higher self versus external assessment scores, whereas 3 studies (18%) reported lower self versus external assessment scores. Sixteen studies (94%) focused on retrospective self-assessment and 1 study (6%) focused on predictive self-assessment. Agreement improved with higher levels of participant training; with high-quality, timely, and relevant feedback; and with postprocedure video review.

CONCLUSIONS

This review demonstrated mixed results regarding an agreement between self and external assessment scores for technical tasks in surgery. Future investigations should attempt to improve the study design by accounting for differences between men and women, conducting paired and independent mean comparisons of self and external assessments, and ensuring that external assessments are valid and reliable.

Surgical Videos for Accident Analysis, Performance Improvement, and Complication Prevention

Conventional audit of surgical records through review of surgical results provides useful knowledge but hardly helps identify the technical reasons lying behind specific outcomes or complications. Surgical teams not only need to know that a complication might happen but also how and when it is most likely to happen. Functional awareness is therefore needed to prevent complications, know how to deal with them, and improve overall surgical performance. The authors wish to argue that the systematic recording and reviewing of surgical videos, a “surgical black box,” might improve surgical care, help prevent complications, and allow accident analysis. A possible strategy to test this hypothesis is presented and discussed. Recording and reviewing surgical interventions, apart from helping us achieve functional awareness and increasing the safety profile of our performance, allows us also to effectively share our experience with colleagues. The authors believe that those potential implications make this hypothesis worth testing.