Objective assessment of total laparoscopic hysterectomy: Development and validation of a feasible rating scale for formative and summative feedback

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review

INTRODUCTION

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity.

MATERIAL AND METHODS

A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis.

RESULTS

A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes.

CONCLUSIONS

Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

Development of a formative feedback tool for transanal total mesorectal excision

INTRODUCTION

Transanal total mesorectal excision (TaTME) is a novel procedure in the treatment of rectal cancer. Current training models for TaTME suggest a period of proctored cases, but no structured feedback tool exists to guide operators during the learning phase. The objective of this study therefore was to develop a formative feedback tool for the critical steps of the transanal portion of TaTME.

METHODS

A two-round Delphi study by TaTME experts was conducted to determine the items to be included in the formative feedback tool. Participants rated each step from a prepared list using a Likert scale from 1 (Not relevant) to 5 (Very relevant) with the option to suggest additional steps. Responses to the first round were presented in the second round, where participants rated the revised list of steps. Consensus was defined as > 80% of participants rating the step as 4 or 5 (out of 5). Items were combined when appropriate to avoid redundancy. Rating anchors describing performance (on a 5-point scale) were then developed for each step. The final tool was recirculated and participants rated the finished product on its feasibility and usefulness.

RESULTS

Twenty-six TaTME experts were contacted for participation. Fifteen experts (58%) participated in the first round of the study, and eleven (42%) participated in the second round. The majority (14, 93%) had completed fellowship training in colorectal surgery. The first round of the Delphi study contained 34 items, and 32 items met inclusion criteria after the second round. Redundant items were combined into 15 items that comprised the final tool. Out of eight respondents to the feasibility survey, all believed the feedback tool enhances the feedback of learners and would use it for training purposes if available.

CONCLUSION

This work describes the development of a novel consensus-based formative feedback tool specific to TaTME.

Tools for measuring technical skills during gynaecologic surgery: a scoping review

BACKGROUND

Standardised assessment is key to structured surgical training. Currently, there is no consensus on which surgical assessment tool to use in live gynaecologic surgery. The purpose of this review is to identify assessment tools measuring technical skills in gynaecologic surgery and evaluate the measurement characteristics of each tool.

METHOD

We utilized the scoping review methodology and searched PubMed, Medline, Embase and Cochrane. Inclusion criteria were studies that analysed assessment tools in live gynaecologic surgery. Kane's validity argument was applied to evaluate the assessment tools in the included studies.

RESULTS

Eight studies out of the 544 identified fulfilled the inclusion criteria. The assessment tools were categorised as global rating scales, global and procedure rating scales combined, procedure-specific rating scales or as a non-procedure-specific error assessment tool.

CONCLUSION

This scoping review presents the current different tools for observational assessment of technical skills in intraoperative, gynaecologic surgery. This scoping review can serve as a guide for surgical educators who want to apply a scale or a specific tool in surgical assessment.

Performance Assessment for Total Laparoscopic Hysterectomy in the Operating Room: Validity Evidence of a Procedure-specific Rating Scale

STUDY OBJECTIVE

The technical conduct of total laparoscopic hysterectomy (LH) is critical to surgical outcomes. This study explored the validity evidence of an objective scale specific to the assessment of technical skills (H-OSATS) for 7 tasks of an LH with salpingo-oophorectomy procedure performed in the operating room.

DESIGN

Observational cohort study.

SETTING

Two academic hospitals in Marseille and Montpellier, France.

PATIENTS

Three groups of operators (novice, intermediate, and experienced surgeons) were video recorded during their live performances of LH on a simple case. For each group, a dozen unedited videos were obtained for the following tasks: division of the round ligament, division of the infundibulopelvic ligament, creation of the bladder flap, opening of the posterior peritoneum, division of the uterine vessels, colpotomy, and closure of the vault.

INTERVENTIONS

Two qualified raters blindly assessed each video using the H-OSATS rating scale. Inter-rater reliability and test-retest reliability were calculated as measures of internal structure. In a separate round of evaluations, the raters provided a global competent/noncompetent decision for each performance. As a measure of consequential validity, a pass/fail score was set for each task using the contrasting group method.

MEASUREMENTS AND MAIN RESULTS

Three tasks (creation of the bladder flap, colpotomy, and closure of the vault) displayed sound validity evidence: a meaningful total score difference among the 3 groups of experience as well as between the intermediate and experienced surgeons, reliability outcomes of >0.7, and a pass/fail score with a theoretical false-positive rate of <10%.

CONCLUSION

The validity evidence of the H-OSATS rating scale differed for separate evaluations of the 7 tasks. Three tasks (i.e., creation of the bladder flap, colpotomy, and closure of the vault) revealed sound validity evidence, including at the level of the attending surgeon, whereas other tasks were more consistent with low-stakes formative evaluation standards.

Systematic Review of Intraoperative Assessment Tools in Minimally Invasive Gynecologic Surgery

OBJECTIVE

To collect, summarize, and evaluate the currently available intraoperative rating tools used in abdominal minimally invasive gynecologic surgery (MIGS).

DATA SOURCES

Medline, Embase, and Scopus databases from January 1, 2000, to May 12, 2020.

METHODS OF STUDY SELECTION

A systematic search strategy was designed and executed. Published studies evaluating an assessment tool in abdominal MIGS cases were included. Studies focused on simulation, reviews, and abstracts without a published manuscript were excluded. Risk of bias and methodological quality were assessed for each study.

TABULATION, INTEGRATION, AND RESULTS

Disparate study methods prevented quantitative synthesis of the data. Ten studies were included in the analysis. The tools were grouped into global (n = 4) and procedure-specific assessments (n = 6). Most studies evaluated small numbers of surgeons and lacked a comparison group to evaluate the effectiveness of the tool. All studies demonstrated content validity and at least 1 dimension of reliability, and 2 have external validity. The intraoperative procedure-specific tools have been more thoroughly evaluated than the global scales.

CONCLUSION

Procedure-specific intraoperative assessment tools for MIGS cases are more thoroughly evaluated than global tools; however, poor-quality studies and borderline reliability limit their use. Well-designed, controlled studies evaluating the effectiveness of intraoperative assessment tools in MIGS are needed.

Laparoscopic Versus Robotic-assisted Suturing Performance Among Novice Surgeons: A Blinded, Cross-Over Study

Robotic-assisted laparoscopy (RAL) presents several advantages over 3-dimensional conventional laparoscopy (3D-CL) that may facilitate laparoscopic suturing especially with novice surgeons. This study compares novice surgeons' suturing performance by 3D-CL and RAL using Objective Structured Assessment of Technical Skill (OSATS), an objective, validated scoring tool. Twenty-two surgeons with no robotic experience completed a standardized suturing task in an experimental setup by both 3D-CL and RAL in a randomized, cross-over design. Two experienced surgeons blindly assessed their performance using OSATS. Median (interquartile range) OSATS scores for 3D-CL and RAL were, respectively, 22.8 (17.4 to 25.8) versus 25.0 (21.9 to 26.5), P=0.032. There was no association between laparoscopic experience and robotic-assisted suturing performance. Thus, this study is, to our knowledge, the first to compare novice surgeons' suturing performance by 3D-CL and RAL using an objective, validated scoring tool and to show better suturing performance when assisted by the robot regardless of experience level.