Technical Performance as a Predictor of Clinical Outcomes in Laparoscopic Gastric Cancer Surgery

Appraisal of surgical outcomes and oncological efficiency of intraoperative adverse events in robotic radical gastrectomy for gastric cancer

BACKGROUND

Surgical quality control is a crucial determinant of evaluating the tumor efficacy.

OBJECTIVE

To assess the ClassIntra grade for quality control and oncological outcomes of robotic radical surgery for gastric cancer (GC).

METHODS

Data of patients undergoing robotic radical surgery for GC at a high-volume center were retrospectively analyzed. Patients were categorized into two groups, the intraoperative adverse event (iAE) group and the non-iAE group, based on the occurrence of intraoperative adverse events. The iAEs were further classified into five sublevels (ranging from I to V according to severity) based on the ClassIntra grade. Surgical performance was assessed using the Objective Structured Assessment of Technical Skill (OSATS) and the General Error Reporting Tool.

RESULTS

This study included 366 patients (iAE group: n = 72 [19.7%] and non-iAE group: n = 294 [80.3%]). The proportion of ClassIntra grade II patients was the highest in the iAE group (54.2%). In total and distal gastrectomies, iAEs occurred most frequently in the suprapancreatic area (50.0% and 54.8%, respectively). In total gastrectomy, grade IV iAEs were most common during lymph node dissection in the splenic hilum area (once for bleeding [grade IV] and once for injury [grade IV]). The overall survival (OS) and disease-free survival of the non-iAE group were significantly better than those of the iAE group (Log rank P < 0.001). Uni- and multi-variate analyses showed that iAEs were key prognostic indicators, independent of tumor stage and adjuvant chemotherapy (P < 0.001).

CONCLUSION

iAEs in patients who underwent robotic radical gastrectomy significantly correlated with the occurrence of postoperative complications and a poor long-term prognosis. Therefore, utilization and inclusion of ClassIntra grading as a crucial surgical quality control and prognostic indicator in the routine surgical quality evaluation system are recommended.

Nationwide Association of Surgical Performance of Minimally Invasive Esophagectomy With Patient Outcomes

Importance

Suboptimal surgical performance is hypothesized to be associated with less favorable patient outcomes in minimally invasive esophagectomy (MIE). Establishing this association may lead to programs that promote better surgical performance of MIE and improve patient outcomes.

Objective

To investigate associations between surgical performance and postoperative outcomes after MIE.

Design, Setting, and Participants

In this nationwide cohort study of 15 Dutch hospitals that perform more than 20 MIEs per year, 7 masked expert MIE surgeons assessed surgical performance using videos and a previously developed and validated competency assessment tool (CAT). Each hospital submitted 2 representative videos of MIEs performed between November 4, 2021, and September 13, 2022. Patients registered in the Dutch Upper Gastrointestinal Cancer Audit between January 1, 2020, and December 31, 2021, were included to examine patient outcomes.

Exposure

Hospitals were divided into quartiles based on their MIE-CAT performance score. Outcomes were compared between highest (top 25%) and lowest (bottom 25%) performing quartiles. Transthoracic MIE with gastric tube reconstruction.

Main Outcome and Measure

The primary outcome was severe postoperative complications (Clavien-Dindo ≥3) within 30 days after surgery. Multilevel logistic regression, with clustering of patients within hospitals, was used to analyze associations between performance and outcomes.

Results

In total, 30 videos and 970 patients (mean [SD] age, 66.6 [9.1] years; 719 men [74.1%]) were included. The mean (SD) MIE-CAT score was 113.6 (5.5) in the highest performance quartile vs 94.1 (5.9) in the lowest. Severe postoperative complications occurred in 18.7% (41 of 219) of patients in the highest performance quartile vs 39.2% (40 of 102) in the lowest (risk ratio [RR], 0.50; 95% CI, 0.24-0.99). The highest vs the lowest performance quartile showed lower rates of conversions (1.8% vs 8.9%; RR, 0.21; 95% CI, 0.21-0.21), intraoperative complications (2.7% vs 7.8%; RR, 0.21; 95% CI, 0.04-0.94), and overall postoperative complications (46.1% vs 65.7%; RR, 0.54; 95% CI, 0.24-0.96). The R0 resection rate (96.8% vs 94.2%; RR, 1.03; 95% CI, 0.97-1.05) and lymph node yield (mean [SD], 38.9 [14.7] vs 26.2 [9.0]; RR, 3.20; 95% CI, 0.27-3.21) increased with oncologic-specific performance (eg, hiatus dissection, lymph node dissection). In addition, a high anastomotic phase score was associated with a lower anastomotic leakage rate (4.6% vs 17.7%; RR, 0.14; 95% CI, 0.06-0.31).

Conclusions and Relevance

These findings suggest that better surgical performance is associated with fewer perioperative complications for patients with esophageal cancer on a national level. If surgical performance of MIE can be improved with MIE-CAT implementation, substantially better patient outcomes may be achievable.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Importance

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

Objective

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

Design, Setting, and Participants

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

Exposure

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

Main Outcomes and Measures

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

Results

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

Conclusions and Relevance

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety

BACKGROUND

Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric.

METHODS

The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2.

RESULTS

The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases.

CONCLUSIONS

A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.

OSATS scoring confirms ICG enhancement of performance in laparoscopic radical gastrectomy: a post-hoc analysis of a randomized controlled trial

BACKGROUND

Indocyanine green (ICG) fluorescence imaging is effective in increasing the number of lymph node dissections during laparoscopic radical gastrectomy; however, no studies have attempted to explain this phenomenon.

METHODS

This study utilized the data from a previous randomized controlled trial (FUGES-012 study) investigating ICG-guided laparoscopic radical gastrectomy performed between November 2018 and July 2019. The Objective Structured Assessments of Technical Skills (OSATS) scoring system was used to grade videos from the ICG and non-ICG groups. Patients with an OSATS score greater than 29 were classified as the high-OSATS population, while those with an OSATS score less than or equal to 29 were classified as the low-OSATS population.

RESULTS

A total of 258 patients were included in the modified intention-to-treat analysis: 129 in the ICG group and 129 in the non-ICG group. The OSATS score of the ICG group was higher than that of the non-ICG group (29.6±2.6 vs. 26.6±3.6; P <0.001). The ICG group underwent a significantly higher mean total number of lymph node dissections than the non-ICG group (50.5±15.9 vs. 42.0±10.3; adjusted P <0.001). The group assigned to ICG use, better OSATS (high-OSATS) scores were observed, which correlated with greater D2 lymph node retrieval (54.1±15.0 vs. 47.2±8.7; adjusted P =0.039). Finally, the ICG group had a lower rate of lymph node noncompliance than that of the non-ICG group (31.8 vs. 57.4%; P <0.001).

CONCLUSIONS

By applying the ICG fluorescence navigation technique, better OSATS scores were observed, which correlated with greater lymph node retrieval and a lower lymph node noncompliance rate, as recommended for individualized laparoscopic radical gastrectomy.

Evidence-Based Surgery: What Can Intra-Operative Images Contribute?

Evidence-based medicine integrates results from randomized controlled trials (RCTs) and meta-analyses, combining the best external evidence with individual clinical expertise and patients' preferences. However, RCTs of surgery differ from those of medicine in that surgical performance is often assumed to be consistent. Yet, evaluating whether each surgery is performed to the same standard is quite challenging. As a primary issue, the novelty of this review is to emphasize-with a focus on orthopedic trauma-the advantage of having complete intra-operative image documentation, allowing the direct evaluation of the quality of the intra-operative technical performance. The absence of complete intra-operative image documentation leads to the inhomogeneity of case series, yielding inconsistent results due to the impossibility of a secondary analysis. Thus, comparisons and the reproduction of studies are difficult. Access to complete intra-operative image data in surgical RCTs allows not only secondary analysis but also comparisons with similar cases. Such complete data can be included in electronic papers. Offering these data to peers-in an accessible link-when presenting papers facilitates the selection process and improves publications for readers. Additionally, having access to the full set of image data for all presented cases serves as a rich resource for learning. It enables the reader to sift through the information and pinpoint the details that are most relevant to their individual needs, allowing them to potentially incorporate this knowledge into daily practice. A broad use of the concept of complete intra-operative image documentation is pivotal for bridging the gap between clinical research findings and real-world applications. Enhancing the quality of surgical RCTs would facilitate the equalization of evidence acquisition in both internal medicine and surgery. Joint effort by surgeons, scientific societies, publishers, and healthcare authorities is needed to support the ideas, implement economic requirements, and overcome the mental obstacles to its realization.

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

BACKGROUND

Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery.

METHODS

A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of the included studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB.

CONCLUSION

AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.

Assessing Chest Tube Insertion Skills Using a Porcine Rib Model-A Validity Study

INTRODUCTION

Assessments require sufficient validity evidence before their use. The Assessment for Competence in Chest Tube Insertion (ACTION) tool evaluates proficiency in chest tube insertion (CTI), combining a rating scale and an error checklist. The aim of this study was to collect validity evidence for the ACTION tool on a porcine rib model according to the Messick framework.

METHODS

A rib model, consisting of a porcine hemithorax that was placed in a wooden frame, was used as simulator. Participants were recruited from the departments of surgery, pulmonology, and emergency medicine. After familiarization with the rib model and the equipment, standardized instructions and clinical context were provided. They performed 2 CTIs while being scored with the ACTION tool. All performances were assessed live by 1 rater and by 3 blinded raters using video recordings. Generalizability-analysis was performed and mean scores and errors of both groups on the first performance were compared. A pass/fail score was established using the contrasting groups' method.

RESULTS

Nine novice and 8 experienced participants completed the study. Generalizability coefficients where high for the rating scale (0.92) and the error checklist (0.87). In the first CTI, novices scored lower than the experienced group (38.1/68 vs. 47.1/68, P = 0.042), but no difference was observed on the error checklist. A pass/fail score of 44/68 was established.

CONCLUSION

A solid validity argument for the ACTION tool's rating scale on a porcine rib model is presented, allowing formative and summative assessment of procedural skills during training before patient contact.

Preserving privacy in surgical video analysis using a deep learning classifier to identify out-of-body scenes in endoscopic videos

Surgical video analysis facilitates education and research. However, video recordings of endoscopic surgeries can contain privacy-sensitive information, especially if the endoscopic camera is moved out of the body of patients and out-of-body scenes are recorded. Therefore, identification of out-of-body scenes in endoscopic videos is of major importance to preserve the privacy of patients and operating room staff. This study developed and validated a deep learning model for the identification of out-of-body images in endoscopic videos. The model was trained and evaluated on an internal dataset of 12 different types of laparoscopic and robotic surgeries and was externally validated on two independent multicentric test datasets of laparoscopic gastric bypass and cholecystectomy surgeries. Model performance was evaluated compared to human ground truth annotations measuring the receiver operating characteristic area under the curve (ROC AUC). The internal dataset consisting of 356,267 images from 48 videos and the two multicentric test datasets consisting of 54,385 and 58,349 images from 10 and 20 videos, respectively, were annotated. The model identified out-of-body images with 99.97% ROC AUC on the internal test dataset. Mean ± standard deviation ROC AUC on the multicentric gastric bypass dataset was 99.94 ± 0.07% and 99.71 ± 0.40% on the multicentric cholecystectomy dataset, respectively. The model can reliably identify out-of-body images in endoscopic videos and is publicly shared. This facilitates privacy preservation in surgical video analysis.

The effect of bedside assistant technical performance on outcomes in robotic surgery

Technical performance in surgery has been associated with patient outcomes. Robotic surgery is unique in that both a console surgeon and bedside surgeon are required. A systematic review according to PRISMA guidelines identified all pertinent literature regarding skill level of the bedside assistant with regards to patient outcomes in robotic surgery. 10 studies met inclusion criteria. In all studies, the skill level of the assistant was based on experience, either by post-graduate year of the resident or number of cases previously performed by the assistant. Five studies reported significant, shorter operative times with increasing experience of the bedside assistant. No study reported a significant difference in postoperative outcomes. The existing literature fails to show improved patient outcomes with more experienced bedside assistants in robotic surgery. Metrics should be developed to measure actual technical performance of the bedside assistant rather than using arbitrary assessments of experience in future studies.

Factors affecting the quality of laparoscopic D2 lymph node dissection for gastric cancer: a cohort study from two randomized controlled trials

BACKGROUND

Laparoscopic D2 lymph node dissection (LND) for gastric cancer has a wide range and high difficulty. In the past, the quality of surgery was often judged by the time of the operation or the amount of blood loss, but the analysis based on surgical video was rarely reported. The purpose of this study was to analyze the relationship between the quality of laparoscopic D2 LND for gastric cancer and postoperative complications.

METHODS

The surgical video and clinicopathological data of 610 patients in two randomized controlled trials in our center from 2013 to 2016 were retrospectively analyzed. Klass-02-QC LND scale and general error score tool were used to quantitatively evaluate the intraoperative performance of D2 LND. Logistic regression was used to analyze the influencing factors of postoperative complications.

RESULTS

The overall incidence of complications (CD classification≥2) was 20.6%; the incidence of surgical complications was 6.9%. According to whether the LND score reached 44, patients were divided into a qualified group (73%) and a not-qualified group (27%). Event score (ES) by quartile was divided into grade 1 (21.7%), grade 2 (26%), grade 3 (28%), and grade 4 (24.3%) from low to high. Univariate logistic regression analysis showed that ES greater than or equal to 3, tumor size greater than or equal to 35 mm, and cTNM >II were independent risk factors for not-qualified LND. Male,tumor size greater than or equal to 35 mm and cTNM >II were independent risk factors for grade 4 ES. Not-qualified LND (OR=1.62, 95% CI: 1.16-3.89, P =0.021), grade 4 ES (OR=3.21, 95% CI: 1.52-3.90, P =0.035), and cTNM >II (OR=1.74, 95% CI: 1.39-7.33, P =0.041) were independent risk factors for postoperative surgical complications.

CONCLUSIONS

The qualification of LND and intraoperative events based on surgical video are the independent influencing factors of postoperative complications of laparoscopic gastric cancer surgery. Specialist training and teaching based on surgical video may help to improve the surgical skills of specialists and improve the postoperative outcome of patients.

Intraoperative performance and outcomes of robotic and laparoscopic total gastrectomy for gastric cancer: A high-volume center retrospective propensity score matching study

BACKGROUND

Studies on robotic total gastrectomy (RTG) are currently limited. This study aimed to compare the intraoperative performance as well as short- and long-term outcomes of RTG and laparoscopic total gastrectomy (LTG).

METHODS

A total of 969 patients underwent robotic (n = 161) or laparoscopic (n = 636) total gastrectomy between October 2014 and October 2021. The two groups of patients were matched 1:3 using the propensity score matching (PSM) method. The intraoperative performance as well as short- and long-term outcomes of the robotic (n = 147) and the laparoscopic (n = 371) groups were compared.

RESULTS

After matching, the estimated intraoperative blood loss was lower (80.51 ± 68.77 vs. 89.89 ± 66.12, p = 0.008), and the total number of lymph node dissections was higher (34.74 ± 12.44 vs. 29.83 ± 12.22, p < 0.001) in the RTG group compared with the LTG group. More lymph node dissections at the upper edge of the pancreas were performed in the RTG group than in the LTG (12.59 ± 4.18 vs. 10.33 ± 4.58, p = 0.001). Additionally, postoperative recovery indicators and laboratory data were greater in the RTG group than those in the LTG group, while postoperative complications were comparable between the two groups (19.0% vs. 18.9%, p = 0.962). For overweight or obese patients with body mass indexes (BMIs) ≥25, certain clinical outcomes of the RTG remained advantageous, and no significant differences in three-year overall survival (OS) or relapse-free survival (RFS) were observed.

CONCLUSIONS

Robotic total gastrectomy demonstrated better intraoperative performance, could improve the short-term clinical outcomes of patients, and was more conducive to patient recovery. However, the long-term efficacies of the two approaches were similar. Robotic surgical systems may reduce surgical stress responses in patients, allowing them to receive postoperative chemotherapy sooner.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery

OBJECTIVE

The American Board of Surgery (ABS) sought to investigate the suitability of video-based assessment (VBA) as an adjunct to certification for assessing technical skills.

BACKGROUND

Board certification is based on the successful completion of a residency program coupled with knowledge and reasoning assessments. VBA is a new modality for evaluating operative skills that have been shown to correlate with patient outcomes after surgery.

METHODS

Diplomates of the ABS were initially assessed for background knowledge and interest in VBA. Surgeons were then solicited to participate in the pilot. Three commercially available VBA platforms were identified and used for the pilot assessment. All participants served as reviewers and reviewees for videos. After the interaction, participants were surveyed regarding their experiences and recommendations to the ABS.

RESULTS

To the initial survey, 4853/25,715 diplomates responded. The majority were neither familiar with VBA, nor the tools used for operative assessments. Two hundred seventy-four surgeons actively engaged in the subsequent pilot. One hundred sixty-nine surgeons completed the postpilot survey. Most participants found the process straightforward. Of the participants, 74% felt that the feedback would help their surgical practice. The majority (81%) remain interested in VBA for continuing medical education credits. Using VBA in continuous certification could improve surgeon skills felt by 70%. Two-thirds of participants felt VBA could help identify and remediate underperforming surgeons. Identified barriers to VBA included limitations for open surgery, privacy issues, and technical concerns.

CONCLUSIONS

VBA is promising as an adjunct to the current board certification process and should be further considered by the ABS.

Validation study of a skill assessment tool for education and outcome prediction of laparoscopic distal gastrectomy

BACKGROUND

The Japanese operative-rating scale for laparoscopic distal gastrectomy (JORS-LDG) was developed through cognitive task analysis together with the Delphi method to measure intraoperative performance during laparoscopic distal gastrectomy. This study aimed to investigate the value of this rating scale as an educational tool and a surgical outcome predictor in laparoscopic distal gastrectomy.

METHODS

The surgical performance of laparoscopic distal gastrectomy was assessed by the first assistant, through self-evaluation in the operating room and by video raters blind to the case. We evaluated inter-rater reliability, internal consistency, and correlations between the JORS-LDG scores and the evaluation methods, patient characteristics, and surgical outcomes.

RESULTS

Fifty-four laparoscopic distal gastrectomy procedures performed by 40 surgeons at 16 institutions were evaluated in the operating room and with video recordings using the proposed rating scale. The video inter-rater reliability was > 0.8. Participating surgeons were divided into the low, intermediate, and high groups based on their total scores. The number of laparoscopic surgeries and laparoscopic gastrectomy procedures performed differed significantly among the groups according to laparoscopic distal gastrectomy skill levels. The low, intermediate, and high groups also differed in terms of median operating times (311, 266, and 229 min, respectively, P < 0.001), intraoperative complication rates (27.8, 11.8, and 0%, respectively, P = 0.01), and postoperative complication rates (22.2, 0, and 0%, respectively, P = 0.002).

CONCLUSIONS

The JORS-LDG is a reliable and valid measure for laparoscopic distal gastrectomy training and could be useful in predicting surgical outcomes.

Surgical instrument detection and tracking technologies: Automating dataset labeling for surgical skill assessment

Surgical skills can be improved by continuous surgical training and feedback, thus reducing adverse outcomes while performing an intervention. With the advent of new technologies, researchers now have the tools to analyze surgical instrument motion to differentiate surgeons’ levels of technical skill. Surgical skills assessment is time-consuming and prone to subjective interpretation. The surgical instrument detection and tracking algorithm analyzes the image captured by the surgical robotic endoscope and extracts the movement and orientation information of a surgical instrument to provide surgical navigation. This information can be used to label raw surgical video datasets that are used to form an action space for surgical skill analysis. Instrument detection and tracking is a challenging problem in MIS, including robot-assisted surgeries, but vision-based approaches provide promising solutions with minimal hardware integration requirements. This study offers an overview of the developments of assessment systems for surgical intervention analysis. The purpose of this study is to identify the research gap and make a leap in developing technology to automate the incorporation of new surgical skills. A prime factor in automating the learning is to create datasets with minimal manual intervention from raw surgical videos. This review encapsulates the current trends in artificial intelligence (AI) based visual detection and tracking technologies for surgical instruments and their application for surgical skill assessment.

Robotic spleen-preserving total gastrectomy shows better short-term advantages: a comparative study with laparoscopic surgery

BACKGROUND

Robotic surgery may be advantageous for complex surgery. We aimed to compare the intraoperative and postoperative short-term outcomes of spleen-preserving splenic hilar lymphadenectomy (SPSHL) during robotic and laparoscopic total gastrectomy.

METHODS

From July 2016 to December 2020, the clinicopathological data of 115 patients who underwent robotic total gastrectomy combined with robotic SPSHL (RSPSHL) and 697 patients who underwent laparoscopic total gastrectomy combined with laparoscopic SPSHL (LSPSHL) were retrospectively analyzed. A 1:2 ratio propensity score matching (PSM) was used to balance the differences between the two groups to compare their outcomes. The Generic Error Rating Tool was used to evaluate the technical performance.

RESULTS

After PSM, the baseline preoperative characteristics of the 115 patients in the RSPSHL and 230 patients in the LSPSHL groups were balanced. The dissection time of the region of the splenic artery trunk (5.4 ± 1.9 min vs. 7.8 ± 3.6 min, P < 0.001), the estimated blood loss during SPSHL (9.6 ± 4.8 ml vs. 14.9 ± 7.8 ml, P < 0.001), and the average number of intraoperative technical errors during SPSHL (15.1 ± 3.4 times/case vs. 20.7 ± 4.3 times/case, P < 0.001) were significantly lower in the RSPSHL group than in the LSPSHL group. The RSPSHL group showed higher dissection rates of No. 10 (78.3% vs. 70.0%, P = 0.104) and No. 11d (54.8% vs. 40.4%, P = 0.012) lymph nodes and significantly improved postoperative recovery results in terms of times to ambulation, first flatus, and first intake (P < 0.05). The splenectomy rates of the two groups were similar (1.7% vs. 0.4%, P = 0.539), and there was no significant difference in morbidity and mortality within postoperative 30 days (13.0% vs. 15.2%, P = 0.589).

CONCLUSION

Compared to LSPSHL, RSPSHL has more advantages in terms of surgical qualities and postoperative recovery process with similar morbidity and mortality. For complex SPSHL, robotic surgery may be a better choice.

The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review

BACKGROUND

Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes.

METHODS

Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis.

RESULTS

A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results.

CONCLUSION

Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.

Use of surgical video-based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study

OBJECTIVE

Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery.

METHODS

Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined-instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions-on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials.

RESULTS

In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001).

CONCLUSIONS

Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.

Correlation of Colorectal Surgical Skill With Patient Outcomes: A Cautionary Tale

BACKGROUND

Previous work has demonstrated a correlation between video ratings of surgical skill and clinical outcomes. Some have proposed the use of video review for technical skill assessment, credentialing, and quality improvement.

OBJECTIVE

Before its adoption as a quality measure for colorectal surgeons, we must first determine whether video-based skill assessments can predict patient outcomes among specialty surgeons.

DESIGN

Twenty-one surgeons submitted one representative video of a minimally invasive colectomy. Each video was edited to highlight key steps and then rated by 10 peer surgeons using a validated American Society of Colon and Rectal Surgeons assessment tool. Linking surgeons' ratings to a validated surgical outcomes registry, we assessed the relationship between skill and risk-adjusted complication rates.

SETTINGS

The study was conducted with the Michigan Surgical Quality Collaborative, a statewide collaborative including 70 community, academic, and tertiary hospitals.

PATIENTS

Patients included those who underwent minimally invasive colorectal resection performed by the participating surgeons.

MAIN OUTCOME MEASURES

Main outcome measures included 30-day risk-adjusted postoperative complications.

RESULTS

The average technical skill rating for each surgeon ranged from 2.6 to 4.6. Risk-adjusted complication rate per surgeon ranged from 9.9% to 33.1%. Patients of surgeons in the bottom quartile of overall skill ratings were older and more likely to have hypertension or to smoke; patients of surgeons in the top quartile were more likely to be immunosuppressed or have an ASA score of 3 or higher. After patient- and surgery-specific risk adjustment, there was no statistically significant difference in complication rates between the bottom and top quartile surgeons (17.5% vs 16.8%, respectively, p = 0.41).

LIMITATIONS

Limitations included retrospective cohort design with short-term follow-up of sampled cases. Videos were edited to highlight key steps, and reviewers did not undergo training to establish norms.

CONCLUSIONS

Our study demonstrates that video-based peer rating of minimally invasive colectomy was not correlated with postoperative complications among specialty surgeons. As such, the adoption of video review for use in credentialing should be approached with caution. See Video Abstract at http://links.lww.com/DCR/B802.CORRELACIÓN ENTRE LA HABILIDAD QUIRÚRGICA COLORRECTAL Y LOS RESULTADOS OBTENIDOS EN EL PACIENTE: RELATO PRECAUTORIOANTECEDENTES:Trabajos anteriores han demostrado una correlación entre la video-calificación de la habilidad quirúrgica y los resultados clínicos. Algunos autores han propuesto el uso de la revisión de videos para la evaluación de la habilidad técnica, la acreditación y la mejoría en la calidad quirúrgica.OBJETIVO:Antes de su adopción como medida de calidad entre los cirujanos colorrectales, primero debemos determinar si las evaluaciones de habilidades basadas en video pueden predecir los resultados clínicos de los pacientes entre cirujanos especializados.DISEÑO:Veintiún cirujanos enviaron un video representativo de una colectomía mínimamente invasiva. Cada video fue editado para resaltar los pasos clave y luego fué calificado por 10 cirujanos revisores utilizando una herramienta de evaluación validada por la ASCRS. Al vincular las calificaciones de los cirujanos al registro de resultados quirúrgicos aprobado, evaluamos la relación entre la habilidad y las tasas de complicaciones ajustadas al riesgo.AJUSTE:Colaboración en todo el estado incluyendo 70 hospitales comunitarios, académicos y terciarios, el Michigan Surgical Quality Collaborative.PACIENTES:Todos aquellos sometidos a resección colorrectal mínimamente invasiva realizada por los cirujanos participantes.MEDIDA DE RESULTADO PRINCIPAL:Complicaciones posoperatorias ajustadas al riesgo a los 30 días.RESULTADOS:La calificación de la habilidad técnica promedio de cada cirujano osciló entre 2.6 y 4.6. La tasa de complicaciones ajustada al riesgo por cirujano osciló entre el 9,9% y el 33,1%. Los pacientes operados por los cirujanos del cuartil inferior de las calificaciones generales de habilidades eran fumadores y añosos, y tambiés más propensos a la hipertensión arterial. Los pacientes operados por los cirujanos del cuartil superior tenían más probabilidades de ser inmunosuprimidos o tener una puntuación ASA> = 3. Después del ajuste de riesgo específico de la cirugía y el paciente, no hubo diferencias estadísticamente significativas en las tasas de complicaciones entre los cirujanos del cuartil inferior y superior (17,5% frente a 16,8%, respectivamente, p = 0,41).LIMITACIONES:Diseño de cohortes retrospectivo con seguimiento a corto plazo de los casos muestreados. Los videos se editaron para resaltar los pasos clave y los revisores no recibieron capacitación para establecer normas.CONCLUSIONES:Nuestro estudio demuestra que la evaluación realizada por los revisores basada en el video de la colectomía mínimamente invasiva no se correlacionó con las complicaciones post-operatorias entre los cirujanos especialistas. Por tanto, la adopción de la revisión del video quirúrgico para su uso en la acreditación profesional, debe abordarse con mucha precaución. Consulte Video Resumen en http://links.lww.com/DCR/B802. (Traducción-Dr. Xavier Delgadillo).

Developing a longitudinal database of surgical skills performance for practicing surgeons: A formal feasibility and acceptance inquiry

BACKGROUND

We explored the feasibility and surgeons' perceptions of the utility of a longitudinal skills performance database.

METHODS

A 10-station surgical skills assessment center was established at a national scientific meeting. Skills assessment volunteers (n = 189) completed a survey including opinions on practicing surgeons' skills evaluation, ethics, and interest in a longitudinal database. A subset (n = 23) participated in a survey-related interview.

RESULTS

Nearly all participants reported interest in a longitudinal database and most believed there is an ethical obligation for such assessments to protect the public. Several interviewees specified a critical role for both formal and informal evaluation is to first create a safe and supportive environment.

CONCLUSIONS

Participants support the construction of longitudinal skills databases that allow information sharing and establishment of professional standards. In a constructive environment, structured peer feedback was deemed acceptable to enhance and diversify surgeon skills. Large scale skills testing is feasible and scientific meetings may be the ideal location.

Learning a MIPO technique for distal radius fractures: Mentoring versus simple experience versus deliberate practice

BACKGROUND

Performance skills of a surgeon for a specific surgical technique range from 1 (novice) to 5 (expert). Mentoring can bring the surgeon to level 3. Simple experience rarely allows achievement of level 5, in contrast to deliberate practice, in which performance skills are improved by setting learning goals based on feedback about the trainee's previous performance.

HYPOTHESIS

When learning a technique for the internal fixation of distal radius fractures, the level of performance skills achieved is higher with deliberate practice than with mentoring or simple experience.

MATERIAL AND METHODS

Four surgeons each performed minimally invasive plate osteosynthesis (MIPO) of 15 distal radius fractures. The procedures were recorded by an HD camera. The first five fractures (step 1) were treated after mentoring, the next 5 fractures after reading an article and viewing a reference video (step 2), and the last 5 fractures after viewing and commenting 20 short videos highlighting possible errors (step 3). Each recording of the procedures performed by the surgeons was scored using the Objective Structured Assessment of Technical Skills (OSATS, 10 items on basic skills [B] and 10 on MIPO-specific skills [S]).

RESULTS

For the basic skills, the mean OSATS scores (on 50) were 31 for step 1, 31 for step 2, and 43 for step 3. For the specific skills, the mean OSATS scores (on 50) were 25 for step 1, 29 for step 2, and 48 for step 3. For overall skills (B+S), the mean OSATS score (on 100) was 56 for step 1, 60 for step 2, and 91 for step 3.

DISCUSSION

Our study confirms that the deliberate practice teaching method is more efficient in improving surgeon skills than simple experience. Deliberate practice consists of four essential steps: (i) setting a well-defined goal; (ii) being motivated to improve one's performance; (iii) receiving immediate feedback; and (iv) having multiple opportunities to repeat and gradually perfect one's performance. Our main hypothesis was verified, since the results of learning a MIPO technique, as assessed using the OSATS scale, were significantly better with a video-assisted deliberate practice technique than with mentoring or simple experience.

CONCLUSION

Video-assisted deliberate practice deserves to be widely used in order to optimise learning curves and to improve risk management in surgery.

LEVEL OF EVIDENCE

III.

Comparative Analysis of Clavien-Dindo Grade and Risk Factors of Complications after Dual-Port Laparoscopic Distal Gastrectomy and Hand-Assisted Laparoscopic Gastrectomy

Objective

To compare the Clavien–Dindo grade and risk factors of complications after dual-port laparoscopic distal gastrectomy (DPLDG) and hand-assisted laparoscopic gastrectomy (HALG).

Methods

The clinical data of 775 patients who underwent DPLDG or HALG in our hospital from May 2016 to May 2019 were retrospectively reviewed, and the patients were divided into the DPLDG group (n = 386) and HALG group (n = 389) according to the surgical method to explore the risk factors of postoperative complications by grading their postoperative complications according to the Clavien–Dindo classification system and single-factor and multivariate analysis of the association between variables in clinical data and complications.

Results

Compared with the HALG group, the DPLDG group had significantly shorter surgical time, less intraoperative blood loss, and better postoperative exhaust time (p < 0.05), with no significant difference in other clinical indicators between the two groups (p > 0.05); the postoperative complication incidence rate of DPLDG group was significantly lower than that of the HALG group; it was shown in the single-factor analysis that the age, tumor length, intraoperative blood loss, pathological stages, and surgical method were related to the postoperative complications, and the results of multivariate analysis indicated that DPLDG was the protective factor for reducing postoperative complications, while age no less than 60 years old and intraoperative blood loss no less than 180 ml were the independent risk factors leading to complications; after surgery, the PNI level values at T1, T2, and T3 of DPLDG group were significantly higher than those of the HALG group (p < 0.05); and at 1 month after surgery, both groups obtained significantly higher GLQI scores than before, and the GLQI score of the DPLDG group was significantly higher in the between-group comparison (p < 0.05).

Conclusion

The DPLDG has lower postoperative complication incidence rate than the HALG, but age no less than 60 years old and intraoperative blood loss not less than 180 ml are the independent risk factors for postoperative complications, so advanced prevention measures shall be taken to lower the incidence of complications.

Clinical Evaluation of Intraoperative Near Misses in Laparoscopic Rectal Cancer Surgery

OBJECTIVE

To investigate the frequency, nature, and severity of intraoperative adverse near miss events within advanced laparoscopic surgery and report any associated clinical impact.

BACKGROUND

Despite implementation of surgical safety initiatives, the intraoperative period is poorly documented with evidence of underreporting. Near miss analyses are undertaken in high-risk industries but not in surgical practice.

METHODS

Case video and data from 2 laparoscopic total mesorectal excision randomized controlled trials were analyzed (ALaCaRT ACTRN12609000663257, 2D3D ISRCTN59485808). Intraoperative adverse events were identified and categorized using the observational clinical human reliability analysis technique. The EAES classification was applied by 2 blinded assessors. EAES grade 1 events (nonconsequential error, no damage, or need for correction) were considered near misses. Associated clinical impact was assessed with early morbidity and histopathology outcomes.

RESULTS

One hundred seventy-five cases contained 1113 error events. Six hundred ninety-eight (62.7%) were near misses (median 3, IQR 2-5, range 0-15) with excellent inter-rater and test-retest reliability (κ=0.86, 95% CI 0.83-0.89, P < 0.001 and κ=0.88, 95% CI 0.85-0.9, P < 0.001 respectively). Significantly more near misses were seen in patients who developed early complications (4 (3-6) vs. 3 (2-4), P < 0.001). Higher numbers of near misses were seen in patients with more numerous (P = 0.002) and more serious early complications (P = 0.003). Cases containing major intraoperative adverse events contained significantly more near misses (5 (3-7) vs. 3 (2-5), P < 0.001) with a major event observed for every 19.4 near misses.

CONCLUSION

Intraoperative adverse events and near misses can be reliably and objectively captured in advanced laparoscopic surgery. Near misses are commonplace and closely associated with morbidity outcomes.

Supporting laparoscopic general surgery training with digital technology: The United Kingdom and Ireland paradigm

Surgical training in the UK and Ireland has faced challenges following the implementation of the European Working Time Directive and postgraduate training reform. The health services are undergoing a digital transformation; digital technology is remodelling the delivery of surgical care and surgical training. This review aims to critically evaluate key issues in laparoscopic general surgical training and the digital technology such as virtual and augmented reality, telementoring and automated workflow analysis and surgical skills assessment. We include pre-clinical, proof of concept research and commercial systems that are being developed to provide solutions. Digital surgical technology is evolving through interdisciplinary collaboration to provide widespread access to high-quality laparoscopic general surgery training and assessment. In the future this could lead to integrated, context-aware systems that support surgical teams in providing safer surgical care.

Automation of surgical skill assessment using a three-stage machine learning algorithm

Surgical skills are associated with clinical outcomes. To improve surgical skills and thereby reduce adverse outcomes, continuous surgical training and feedback is required. Currently, assessment of surgical skills is a manual and time-consuming process which is prone to subjective interpretation. This study aims to automate surgical skill assessment in laparoscopic cholecystectomy videos using machine learning algorithms. To address this, a three-stage machine learning method is proposed: first, a Convolutional Neural Network was trained to identify and localize surgical instruments. Second, motion features were extracted from the detected instrument localizations throughout time. Third, a linear regression model was trained based on the extracted motion features to predict surgical skills. This three-stage modeling approach achieved an accuracy of 87 ± 0.2% in distinguishing good versus poor surgical skill. While the technique cannot reliably quantify the degree of surgical skill yet it represents an important advance towards automation of surgical skill assessment.

Utility of a modified age-adjusted Charlson Comorbidity Index in predicting cause-specific survival among patients with gastric cancer

INTRODUCTION

The current study aimed to evaluate the ability of a modified version of the age-adjusted Charlson Comorbidity Index (mACCI) in predicting cause-specific survival (CSS) among patients with gastric cancer who underwent curative gastrectomy and compared it with the conventional ACCI.

MATERIALS AND METHODS

Patients who underwent gastrectomy for gastric cancer from 2007 to 2016 (n = 2885) were included. A mACCI was established by excluding scores for other malignancies, such as other cancers, leukemia, and lymphoma. After determining the optimal cutoff ACCI and mACCI values for CSS, clinicopathological factors and survival outcomes were assessed according to the ACCI and mACCI.

RESULTS

Both ACCI and mACCI were identified as independent prognostic factors for overall survival (p < 0.001 and p < 0.001, respectively). However, only mACCI was identified as an independent prognostic factor for CSS (p < 0.001). The present study suggested that mACCI was a better indicator of CSS in patients with gastric cancer who underwent curative gastrectomy than ACCI.

CONCLUSION

Our findings showed that the mACCI was a strong predictor of CSS in patients with gastric cancer who underwent curative gastrectomy. We believe that the mACCI will become a novel marker that would guide treatment decisions for patients with gastric cancer suffering from comorbidities.

Educational application of intraoperative records from an energy device in laparoscopic gastrectomy: a preliminary report

PURPOSE

Video review is a reliable method for surgical education in laparoscopic gastrectomy (LG), but more objective methods are still needed. The purpose of this study was to determine whether the energy device records reflected surgical competency, and thereby may improve surgical education.

METHODS

A total of 16 patients who underwent LG for gastric cancer using the Thunderbeat^® device were preliminarily retrospectively analyzed. This device has the function of 'intelligent tissue monitoring' (ITM), a safety assist system stopping energy output, and can record ITM detections and firing time during surgery. The number of ITM detections and firings, and the total firing time during gastrocolic ligament dissection and infrapyloric dissection were compared between trainees (n = 9 by 5 surgeons) and experts (n = 7 by 5 surgeons). The non-edited videos (n = 16) were scored, and the correlations between the scores and the records were then analyzed.

RESULTS

Significantly more ITM detections, firings, and a longer total firing time were observed in trainees than in experts. The number of ITM detections and firing had negative correlations with the scores of the operation speed, the use of the non-dominant hand, and the use of an energy device.

CONCLUSIONS

Our preliminary study suggested that the above described energy device records reflected surgical competency, and thereby may improve surgical education.

Development and Evaluation of a Novel Instrument to Measure Severity of Intraoperative Events Using Video Data

OBJECTIVE

To develop and evaluate a novel instrument to measure SEVERE processes using video data.

BACKGROUND

Surgical video data can serve an important role in understanding the relationship between intraoperative events and postoperative outcomes. However, a standard tool to measure severity of intraoperative events is not yet available.

METHODS

Items to be included in the instrument were identified through literature and video reviews. A committee of experts guided item reduction, including pilot tests and revisions, and determined weighted scores. Content validity was evaluated using a validated sensibility questionnaire. Inter-rater reliability was assessed by calculating intraclass correlation coefficient. Construct validity was evaluated on a sample of 120 patients who underwent laparoscopic Roux-en-Y gastric bypass procedure, in which comprehensive video data was obtained.

RESULTS

SEVERE index measures severity of 5 event types using ordinal scales. Each intraoperative event is given a weighted score out of 10. Inter-rater reliability was excellent [0.87 (95%-confidence interval, 0.77-0.92)]. In a sample of consecutive 120 patients undergoing gastric bypass procedures, a median of 12 events [interquartile range (IQR) 9-18] occurred per patient and bleeding was the most frequent type (median 10, IQR 7-14). The median SEVERE score per case was 11.3 (IQR 8.3-16.9). In risk-adjusted multivariable regression models, history of previous abdominal surgery (P = 0.02) and body mass index (P = 0.005) were associated with SEVERE scores, demonstrating construct validity evidence.

CONCLUSION

The SEVERE index may prove to be a useful instrument in identifying patients with high risk of developing postoperative complications.

The what? How? And Who? Of video based assessment

BACKGROUND

Currently, there is significant variability in the development, implementation and overarching goals of video review for assessment of surgical performance.

METHODS

This paper evaluates the current methods in which video review is used for evaluation of surgical performance and identifies which processes are critical for successful, widespread implementation of video-based assessment.

RESULTS

Despite the advances in video capture technology and growing interest in video-based assessment, there is a notable gap in the implementation and longitudinal use of formative and summative assessment using video.

CONCLUSION

Validity, scalability and discoverability are current but removable barriers to video-based assessment.

Assessment of 3-Dimensional vs 2-Dimensional Imaging and Technical Performance Using a Multiport Intraoperative Data Capture and Analytic System for Patients Undergoing Laparoscopic Roux-en-Y Gastric Bypass Surgery

IMPORTANCE

Errors and adverse events occur frequently in health care. Three-dimensional (3-D) laparoscopic systems claim to provide more realistic depth perception and better spatial orientation compared with their 2-D counterparts.

OBJECTIVE

To compare the association of 3-D vs 2-D systems with technical performance during laparoscopic Roux-en-Y gastric bypass (LRYGB) procedures using a multiport intraoperative data capture system.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study was performed between May and December 2018, with a total of 50 LRYGB procedures performed in an academic tertiary care center; recordings of the operations were evaluated with a 30-day follow-up. All procedures were performed by the same surgical team.

EXPOSURE

Surgical teams used 2-D or 3-D laparoscopic systems.

MAIN OUTCOMES AND MEASURES

Technical performance was evaluated using the Objective Structured Assessment of Technical Skill and surgical errors and events using the Generic Error Rating Tool.

RESULTS

Of the 50 patients who underwent LRYGB procedures, 42 (86%) were women, with a median (interquartile range) age of 42 (35-47) years and a median (interquartile range) body mass index of 46 (42-48), with no significant demographic differences between the groups whose operations were performed using the 2-D and 3-D systems. The mean (SD) number of errors per case was significantly lower in procedures using the 3-D laparoscopic system than in those using the 2-D system (17 [6] vs 33 [2]; P < .001). The mean (SD) number of error-related events was significantly lower in procedures using the 3-D system than in those using the 2-D system (6 [2] vs 11 [4]; P < .001). Mean (SD) Objective Structured Assessment of Technical Skill scores were significantly higher when the 3-D system was used than when the 2-D system was used (28 [4] vs 22 [3]; P < .001).

CONCLUSIONS AND RELEVANCE

In this limited sample of LRYGB procedures, the use of a 3-D laparoscopic system was associated with a statistically significant reduction in errors and events as well as higher Objective Structured Assessment of Technical Skill scores compared with 2-D systems.

Evidence that surgical performance predicts clinical outcomes

PURPOSE

Assessment of surgeon performance in the operating room has been identified as a direct method of measuring surgical quality. Studies published in urology and other surgical disciplines have investigated this link directly by measuring surgeon and team performance using methodology supported by validity evidence. This article highlights the key findings of these studies and associated underlying concepts.

METHODS

​Seminal literature from urology and related areas of research was used to inform this review of the performance-outcome relationship in surgery. Current efforts to further our understanding of this concept are discussed, including relevant quality improvement and educational interventions that utilize this relationship.

RESULTS

Evidence from multiple surgical specialties and procedures has established the association between surgeon skill and clinically significant patient outcomes. Novel methods of measuring performance utilize surgeon kinematics and artificial intelligence techniques to more reliably and objectively quantify surgical performance.

CONCLUSIONS

Future directions include the use of this data to create interventions for quality improvement, as well as innovate the credentialing and recertification process for practicing surgeons.

Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees

OBJECTIVE

Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video.

DESIGN

Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed.

SETTING

The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital.

PARTICIPANTS

Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard.

RESULTS

The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute.

CONCLUSIONS

Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.

Surgery Tutor for Computational Assessment of Technical Proficiency in Soft-Tissue Tumor Resection in a Simulated Setting

BACKGROUND

In competency-based medical education, progression between milestones requires reliable and valid methods of assessment. Surgery Tutor is an open-source motion tracking platform developed to objectively assess technical proficiency during open soft-tissue tumor resections in a simulated setting. The objective of our study was to provide evidence in support of construct validity of the scores obtained by Surgery Tutor. We hypothesized that Surgery Tutor would discriminate between novice, intermediate, and experienced operators.

METHODS

Thirty participants were assigned to novice, intermediate, or experienced groups, based on the number of prior soft-tissue resections performed. Each participant resected 2 palpable and 2 nonpalpable lesions from a soft-tissue phantom. Surgery Tutor was used to track hand and instrument motions, number of tumor breaches, and time to perform each resection. Mass of excised specimens and margin status were also recorded.

RESULTS

Surgery Tutor scores demonstrated "moderate" to "good" internal structure (test-retest reliability) for novice, intermediate, and experienced groups (interclass correlation coefficient = 0.596, 0.569, 0.737; p < 0.001). Evidence in support of construct validity (consequences) was demonstrated by comparing scores of novice, intermediate, and experienced participantsfor number of hand and instrument motions (690 ± 190, 597 ± 169, 469 ± 110; p < 0.001), number of tumor breaches (29 ± 34, 16 ± 11, 9 ± 6; p < 0.001), time per resection (677 ± 331 seconds, 561 ± 210 seconds, 449 ± 148 seconds; p < 0.001), mass of completely excised specimens (22 ± 7g, 21 ± 11g, 17 ± 6 g; p = 0.035), and rate of positive margin (68%, 50%, 28%; p < 0.001). There was "strong" and "moderate" relationships between motion scores and Objective Structured Assessment of Technical Skill scores, and time per resection and Objective Structured Assessment of Technical Skill scores respectively (r = -0.60, p < 0.001; r = -0.54, p < 0.001).

CONCLUSION

Surgery Tutor scores demonstrate evidenceof construct validity with regards to good internal structure, consequences, and relationship to other variables in the assessment of technical proficiency duringopen soft-tissue tumor resections in a simulated setting. Utilization of Surgery Tutor can provide formative feedback and objective assessment of surgical proficiency in a simulated setting.

Creating Individual Surgeon Performance Assessments in a Statewide Hospital Surgical Quality Improvement Collaborative

BACKGROUND

Surgeon performance profiling is of great interest to surgeons, hospitals, health plans, and the public, yet efforts to date have been contested, with stakeholders at odds over the selection, reliability, and validity of metrics used. We sought to create surgeon-level comparative assessments within the Illinois Surgical Quality Improvement Collaborative.

STUDY DESIGN

American College of Surgeons NSQIP data were obtained for 51 Illinois hospitals covering a 30-month period from 2014 to 2016. Surgeon-level, risk-adjusted outcomes rates were estimated from 3-level crossed random effects logistic regression models and classified as low, as expected, or high for each of 7 postoperative outcomes. Model intra-class correlations and provider-specific reliability statistics were calculated.

RESULTS

A total of 123,141 cases were analyzed for 2,724 surgeons. Median provider case volume was 17 (interquartile range 4 to 54). Overall crude complication rates ranged from 0.62% to 7.14% across the 7 outcomes investigated. Surgeon-level variance estimates were low (intra-class correlation coefficients between 0.007 and 0.074). No performance outliers were detected for 3 of the outcomes measures, while a small number of outliers were identified for any morbidity (11 surgeons), surgical site infection (10 surgeons), death or serious morbidity (8 surgeons), and reoperation (1 surgeon). Among all physicians, median reliability was below 0.1 for each outcome.

CONCLUSIONS

Few individual surgeon performance outliers could be detected in NSQIP clinical registry data for a statewide hospital collaborative over a 30-month period using postoperative patient outcomes. Low surgeon-specific case volumes and minimal variance between surgeons may limit the utility of American College of Surgeons NSQIP outcomes measures for individual profiling. Alternative metrics, such as process measures, patient experience, composite measures, or technical skill assessments should be explored for surgeon-level measurement.