The Effect of Technical Performance on Patient Outcomes in Surgery: A Systematic Review

Study of surgical performance during clavicle plate placements using 2 learning methods: naive practice versus deliberate practice

BACKGROUND

Companionship and simple experience or naive practice (NP) rarely lead to expert level surgery, in contrast to deliberate practice (DP) where an expert analyzes the learner's errors and sets goals to improve performance. The main hypothesis was that using DP for learning would result in faster and/or greater progress than using NP.

OBJECTIVES

The objective of this work was to compare the evolution of the learning curve for clavicle locking plate placement on a sawbone model of a clavicle fracture, by surgical trainees learning via two different methods; NP and DP.

PATIENTS AND METHODS

Ten surgical residents, divided into 2 groups of 5, each placed 6 plates. The 6 trials were filmed. The NP group saw an expert video before each placement. The DP group saw this video once and then received personalized advice from the expert for improvement, by analyzing their own video after each subsequent trial. Objective performance (OP) was measured by a standardized evaluation grid (OSATS, with a score ranging from 10 to 50 points per trial), self-evaluation of performance by a numerical scale (from 0 to 10) and stress by an analgesia-nociception index (ANI, calculated by heart rate recording, from 0 to 100).

RESULTS

The mean OP at the last trial of clavicle plate placement was 41.8 (NP group) and 48.2 (DP group), with a mean progression from the first to last trials of 0.8 in the NP group, and 5.1 in the DP group. The mean progression in self-evaluation between the first and last trials was 3.4 (NP group) and 4.6 (DP group). The mean progression of the ANI between the first and last trials was -4.5 (NP group) and +5 (DP group).

DISCUSSION

The results of learning a clavicle plate osteosynthesis technique measured by OSATS were better with deliberate practice than with naive practice. The progression in self-evaluated performance was better with deliberate practice, but with a higher stress level.

CONCLUSION

Deliberate practice is a technique for learning the surgical procedure which complements companionship and experience. It shortens the learning curve and improves the level of performance of surgical trainees.

LEVEL OF EVIDENCE

IV; non-interventional research.

Capturing relationships between suturing sub-skills to improve automatic suturing assessment

Suturing skill scores have demonstrated strong predictive capabilities for patient functional recovery. The suturing can be broken down into several substep components, including needle repositioning, needle entry angle, etc. Artificial intelligence (AI) systems have been explored to automate suturing skill scoring. Traditional approaches to skill assessment typically focus on evaluating individual sub-skills required for particular substeps in isolation. However, surgical procedures require the integration and coordination of multiple sub-skills to achieve successful outcomes. Significant associations among the technical sub-skill have been established by existing studies. In this paper, we propose a framework for joint skill assessment that takes into account the interconnected nature of sub-skills required in surgery. The prior known relationships among sub-skills are firstly identified. Our proposed AI system is then empowered by the prior known relationships to perform the suturing skill scoring for each sub-skill domain simultaneously. Our approach can effectively improve skill assessment performance through the prior known relationships among sub-skills. Through the proposed approach to joint skill assessment, we aspire to enhance the evaluation of surgical proficiency and ultimately improve patient outcomes in surgery.

Teaching Medical Procedural Skills for Performance

Procedures are a core element of medical professional practice. Today's training approach was formulated in the mid-twentieth century based on a computer analogue of the brain. Despite minor modifications, the system has remained relatively unchanged for the past 70 years. It delivers competence. However, competence is not reliable performance. The inability to adapt to the variety of patients and variations in the performance environments, such as the operating room, results in patient morbidity and mortality. There is a need for changes in the development and training of medical procedural skills based on current theories of skill acquisition, movement theory, and motor control. Achieving optimal performance necessitates the ability to adapt through training in diverse patient and performance environments rather than merely imitating prescribed movements. We propose a novel model of training, the Constraints-Led Approach, which allows for robust training by altering the factors affecting skill acquisition and lifelong learning.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Importance

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

Objective

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

Design, Setting, and Participants

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

Exposure

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

Main Outcomes and Measures

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

Results

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

Conclusions and Relevance

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Development of a Sensor Technology to Objectively Measure Dexterity for Cardiac Surgical Proficiency

BACKGROUND

Technical skill is essential for good outcomes in cardiac surgery. However, no objective methods exist to measure dexterity while performing surgery. The purpose of this study was to validate sensor-based hand motion analysis (HMA) of technical dexterity while performing a graft anastomosis within a validated simulator.

METHODS

Surgeons at various training levels performed an anastomosis while wearing flexible sensors (BioStamp nPoint, MC10 Inc) with integrated accelerometers and gyroscopes on each hand to quantify HMA kinematics. Groups were stratified as experts (n = 8) or novices (n = 18). The quality of the completed anastomosis was scored using the 10 Point Microsurgical Anastomosis Rating Scale (MARS10). HMA parameters were compared between groups and correlated with quality. Logistic regression was used to develop a predictive model from HMA parameters to distinguish experts from novices.

RESULTS

Experts were faster (11 ± 6 minutes vs 21 ± 9 minutes; P = .012) and used fewer movements in both dominant (340 ± 166 moves vs 699 ± 284 moves; P = .003) and nondominant (359 ± 188 moves vs 567 ± 201 moves; P = .02) hands compared with novices. Experts' anastomoses were of higher quality compared with novices (9.0 ± 1.2 MARS10 vs 4.9 ± 3.2 MARS10; P = .002). Higher anastomosis quality correlated with 9 of 10 HMA parameters, including fewer and shorter movements of both hands (dominant, r = -0.65, r = -0.46; nondominant, r = -0.58, r = -0.39, respectively).

CONCLUSIONS

Sensor-based HMA can distinguish technical dexterity differences between experts and novices, and correlates with quality. Objective quantification of hand dexterity may be a valuable adjunct to training and education in cardiac surgery training programs.

The clinical significance of intraoperative adverse events in laparoscopic radical hysterectomies for early-stage cervical cancer

OBJECTIVE

Surgical quality plays a vital role in the treatment of malignant tumors. We investigated the classification of intraoperative adverse events (iAE) (ClassIntra) in relation to the surgical quality control of laparoscopic radical hysterectomies.

METHODS

A prospective cohort of 195 patients who had undergone laparoscopic radical hysterectomies for early stage cervical cancer between July 2019 and July 2021 was enrolled. Participants were classified into either an iAE or non-iAE groups in accordance with their intraoperative status. Surgical outcomes, patient satisfaction, and quality of life were compared between the two groups.

RESULTS

Overall, 48 (24.6%) patients experienced 71 iAE. The iAE group was associated with significantly longer operative times (mean: 270 vs. 245 min, P < 0.001), greater blood loss (mean: 215 vs. 120 mL, P < 0.001), and longer postoperative hospital stays (median: 16 vs. 11 days). Larger tumors and poor technical performance significantly increased the risk of iAE (P < 0.05). Multivariate analysis revealed that iAE were the only independent risk factors for postoperative complications (hazard ratio, 15.100; 95% confidence interval: 4.735-48.158, P < 0.001). Moreover, patients who experienced iAE had significantly lower satisfaction scores and poorer quality of life (P < 0.05).

CONCLUSIONS

ClassIntra may serve as an effective adjunctive tool for surgical quality control in laparoscopic radical hysterectomies.

Development and validation of an interpretable Markov-embedded multilabel model for predicting risks of multiple postoperative complications among surgical inpatients: a multicenter prospective cohort study

BACKGROUND

When they encounter various highly related postoperative complications, existing risk evaluation tools that focus on single or any complications are inadequate in clinical practice. This seriously hinders complication management because of the lack of a quantitative basis. An interpretable multilabel model framework that predicts multiple complications simultaneously is urgently needed.

MATERIALS AND METHODS

The authors included 50 325 inpatients from a large multicenter cohort (2014-2017). The authors separated patients from one hospital for external validation and randomly split the remaining patients into training and internal validation sets. A MARKov-EmbeDded (MARKED) multilabel model was proposed, and three models were trained for comparison: binary relevance, a fully connected network (FULLNET), and a deep neural network. Performance was mainly evaluated using the area under the receiver operating characteristic curve (AUC). The authors interpreted the model using Shapley Additive Explanations. Complication-specific risk and risk source inference were provided at the individual level.

RESULTS

There were 26 292, 6574, and 17 459 inpatients in the training, internal validation, and external validation sets, respectively. For the external validation set, MARKED achieved the highest average AUC (0.818, 95% CI: 0.771-0.864) across eight outcomes [compared with binary relevance, 0.799 (0.748-0.849), FULLNET, 0.806 (0.756-0.856), and deep neural network, 0.815 (0.765-0.866)]. Specifically, the AUCs of MARKED were above 0.9 for cardiac complications [0.927 (0.894-0.960)], neurological complications [0.905 (0.870-0.941)], and mortality [0.902 (0.867-0.937)]. Serum albumin, surgical specialties, emergency case, American Society of Anesthesiologists score, age, and sex were the six most important preoperative variables. The interaction between complications contributed more than the preoperative variables, and formed a hierarchical chain of risk factors, mild complications, and severe complications.

CONCLUSION

The authors demonstrated the advantage of MARKED in terms of performance and interpretability. The authors expect that the identification of high-risk patients and the inference of the risk source for specific complications will be valuable for clinical decision-making.

Variation in Competence of Graduating General Surgery Trainees

OBJECTIVE

To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training.

DESIGN

Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy.

SETTING

This study was conducted using data from members of the SIMPL collaborative.

PARTICIPANTS

A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021.

RESULTS

The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy.

CONCLUSIONS

There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Tissue Acceleration as a Novel Metric for Surgical Performance During Carotid Endarterectomy

BACKGROUND AND OBJECTIVES

Gentle tissue handling to avoid excessive motion of affected fragile vessels during surgical dissection is essential for both surgeon proficiency and patient safety during carotid endarterectomy (CEA). However, a void remains in the quantification of these aspects during surgery. The video-based measurement of tissue acceleration is presented as a novel metric for the objective assessment of surgical performance. This study aimed to evaluate whether such metrics correlate with both surgeons' skill proficiency and adverse events during CEA.

METHODS

In a retrospective study including 117 patients who underwent CEA, acceleration of the carotid artery was measured during exposure through a video-based analysis. Tissue acceleration values and threshold violation error frequencies were analyzed and compared among the surgeon groups with different surgical experience (3 groups: novice , intermediate , and expert ). Multiple patient-related variables, surgeon groups, and video-based surgical performance parameters were compared between the patients with and without adverse events during CEA.

RESULTS

Eleven patients (9.4%) experienced adverse events after CEA, and the rate of adverse events significantly correlated with the surgeon group. The mean maximum tissue acceleration and number of errors during surgical tasks significantly decreased from novice, to intermediate, to expert surgeons, and stepwise discriminant analysis showed that the combined use of surgical performance factors could accurately discriminate between surgeon groups. The multivariate logistic regression analysis revealed that the number of errors and vulnerable carotid plaques were associated with adverse events.

CONCLUSION

Tissue acceleration profiles can be a novel metric for the objective assessment of surgical performance and the prediction of adverse events during surgery. Thus, this concept can be introduced into futuristic computer-aided surgeries for both surgical education and patient safety.

New generation evaluations: video-based surgical assessments : A technology update

BACKGROUND

Surgical skill training, assessment, and feedback are the backbone of surgical training. High-quality skills require expert supervision and evaluation throughout a resource-intensive multi-year training process. As technological barriers to internet access and the ability to save and upload surgical videos continue to improve, video-based assessment technology is emerging as a tool that could reshape surgical training for the next generation of surgeons. Video-based assessment platforms have the potential to allow surgeons from across the globe to upload their surgical videos online and receive high-quality, standardized, and unbiased feedback. They combine visual recordings of a surgeon's operative technique, with standardized grading tools that have the potential to significantly impact surgical training and technical skill acquisition across the world.

METHOD

The platforms included in this review are in various stages of development after a thorough discussion with national experts on the SAGES TAVAC (Technology and Value Assessments) Committee. For each VBA program, a description of its platform was given and a literature review was obtained using a PubMed search performed from inception until December 2021.

RESULTS

The study reviewed all video-based assessment programs currently available in the market, identified their strengths and weaknesses, and how they can be optimized in future.

CONCLUSION

The technological platforms will play a key role in the training and technical skill acquisition of the next generation of surgeons and can have an immense impact on patient care across the world. There is immense potential for all these platforms to grow and become incorporated within the framework of an effective surgical training program.

Pancreatic Adenocarcinoma: Real World Evidence of Care Delivery in AccessHope Data

BACKGROUND

Pancreatic adenocarcinoma is an aggressive disease and the delivery of comprehensive care to individuals with this cancer is critical to achieve appropriate outcomes. The identification of gaps in care delivery facilitates the design of interventions to optimize care delivery and improve outcomes in this population.

METHODS

AccessHope™ is a growing organization that connects oncology subspecialists with treating providers through contracts with self-insured employers. Data from 94 pancreatic adenocarcinoma cases (August 2019-December 2022) in the AccessHope dataset were used to describe gaps in care delivery.

RESULTS

In all but 6% of cases, the subspecialist provided guideline-concordant recommendations anticipated to improve outcomes. Gaps in care were more pronounced in patients with non-metastatic pancreatic cancer. There was a significant deficiency in germline testing regardless of the stage, with only 59% of cases having completed testing. Only 20% of cases were receiving palliative care or other allied support services. There was no difference in observed care gaps between patients receiving care in the community setting vs. those receiving care in the academic setting.

CONCLUSIONS

There are significant gaps in the care delivered to patients with pancreatic adenocarcinoma. A concurrent subspecialist review has the opportunity to identify and address these gaps in a timely manner.

Force Profile as Surgeon-Specific Signature

Objective

To investigate the notion that a surgeon's force profile can be the signature of their identity and performance.

Summary background data

Surgeon performance in the operating room is an understudied topic. The advent of deep learning methods paired with a sensorized surgical device presents an opportunity to incorporate quantitative insight into surgical performance and processes. Using a device called the SmartForceps System and through automated analytics, we have previously reported surgeon force profile, surgical skill, and task classification. However, an investigation of whether an individual surgeon can be identified by surgical technique has yet to be studied.

Methods

In this study, we investigate multiple neural network architectures to identify the surgeon associated with their time-series tool-tissue forces using bipolar forceps data. The surgeon associated with each 10-second window of force data was labeled, and the data were randomly split into 80% for model training and validation (10% validation) and 20% for testing. Data imbalance was mitigated through subsampling from more populated classes with a random size adjustment based on 0.1% of sample counts in the respective class. An exploratory analysis of force segments was performed to investigate underlying patterns differentiating individual surgical techniques.

Results

In a dataset of 2819 ten-second time segments from 89 neurosurgical cases, the best-performing model achieved a micro-average area under the curve of 0.97, a testing F1-score of 0.82, a sensitivity of 82%, and a precision of 82%. This model was a time-series ResNet model to extract features from the time-series data followed by a linearized output into the XGBoost algorithm. Furthermore, we found that convolutional neural networks outperformed long short-term memory networks in performance and speed. Using a weighted average approach, an ensemble model was able to identify an expert surgeon with 83.8% accuracy using a validation dataset.

Conclusions

Our results demonstrate that each surgeon has a unique force profile amenable to identification using deep learning methods. We anticipate our models will enable a quantitative framework to provide bespoke feedback to surgeons and to track their skill progression longitudinally. Furthermore, the ability to recognize individual surgeons introduces the mechanism of correlating outcome to surgeon performance.

Measuring Quality at the Surgeon Level

Patients expect high-quality surgical care and increasingly are looking for ways to assess the quality of the surgeon they are seeing, but quality measurement is often more complicated than one might expect. Measurement of individual surgeon quality in a manner that allows for comparison among surgeons is particularly difficult. While the concept of measuring individual surgeon quality has a long history, technology now allows for new and innovative ways to measure and achieve surgical excellence. However, some recent efforts to make surgeon-level quality data publicly available have highlighted the challenges of this work. Through this chapter, the reader will be introduced to a brief history of surgical quality measurement, learn about the current state of quality measurement, and get a glimpse into what the future holds.

The effect of bedside assistant technical performance on outcomes in robotic surgery

Technical performance in surgery has been associated with patient outcomes. Robotic surgery is unique in that both a console surgeon and bedside surgeon are required. A systematic review according to PRISMA guidelines identified all pertinent literature regarding skill level of the bedside assistant with regards to patient outcomes in robotic surgery. 10 studies met inclusion criteria. In all studies, the skill level of the assistant was based on experience, either by post-graduate year of the resident or number of cases previously performed by the assistant. Five studies reported significant, shorter operative times with increasing experience of the bedside assistant. No study reported a significant difference in postoperative outcomes. The existing literature fails to show improved patient outcomes with more experienced bedside assistants in robotic surgery. Metrics should be developed to measure actual technical performance of the bedside assistant rather than using arbitrary assessments of experience in future studies.

Evolution of the digital operating room: the place of video technology in surgery

PURPOSE

The aim of this review was to collate current evidence wherein digitalisation, through the incorporation of video technology and artificial intelligence (AI), is being applied to the practice of surgery. Applications are vast, and the literature investigating the utility of surgical video and its synergy with AI has steadily increased over the last 2 decades. This type of technology is widespread in other industries, such as autonomy in transportation and manufacturing.

METHODS

Articles were identified primarily using the PubMed and MEDLINE databases. The MeSH terms used were "surgical education", "surgical video", "video labelling", "surgery", "surgical workflow", "telementoring", "telemedicine", "machine learning", "deep learning" and "operating room". Given the breadth of the subject and the scarcity of high-level data in certain areas, a narrative synthesis was selected over a meta-analysis or systematic review to allow for a focussed discussion of the topic.

RESULTS

Three main themes were identified and analysed throughout this review, (1) the multifaceted utility of surgical video recording, (2) teleconferencing/telemedicine and (3) artificial intelligence in the operating room.

CONCLUSIONS

Evidence suggests the routine collection of intraoperative data will be beneficial in the advancement of surgery, by driving standardised, evidence-based surgical care and personalised training of future surgeons. However, many barriers stand in the way of widespread implementation, necessitating close collaboration between surgeons, data scientists, medicolegal personnel and hospital policy makers.

Development and evaluation of a portable and soft 3D-printed cast for laparoscopic choledochojejunostomy model in surgical training

BACKGROUND

Laparoscopic choledochojejunostomy (LCJ) is an essential basic skill for biliary surgeons. Therefore, we established a convenient and effective LCJ 3D printing model to evaluate whether the model could simulate the actual operation situation and determine its effectiveness and validity in surgical training.

METHODS

A 3D printing dry laboratory model was established to simulate LCJ. The face and content validity of the model were evaluated by six experienced biliary surgeons based on 5-point Likert scale questionnaires. A total of 15 surgeons with different levels of experience performed LCJ on the model and evaluated the structural validity of the model using the objective structured assessment of technical skills (OSATS). Simultaneously, the operation time of each surgery was also recorded. A study was also performed to further evaluate the learning curve of residents.

RESULTS

The operating space score of the model was 4.83 ± 0.41 points. The impression score of bile duct and intestinal canal was 4.33 ± 0.52 and 4.17 ± 0.41 points, respectively. The tactile sensation score of bile duct suture and intestinal canal suture was 4.00 ± 0.63 and 3.83 ± 0.41points, respectively. The OSATS score for model operation in the attending group was 29.20 ± 0.45 points, which was significantly higher than that in the fellow group (26.80 ± 1.10, P = 0.007) and the resident group (19.80 ± 1.30, P < 0.001). In addition, there was a statistical difference in operation time among surgeons of different experience levels (P < 0.05). Residents could significantly improve the surgical score and shorten the time of LCJ through repeated training.

CONCLUSIONS

The 3D printing LCJ model can simulate the real operation scenes and distinguish surgeons with different levels of experience. The model is expected to be one of the training methods for biliary tract surgery in the future.

The Sequence of Steps: A Key Concept Missing in Surgical Training-A Systematic Review and Recommendations to Include It

BACKGROUND

Surgical procedures have an inherent feature, which is the sequence of steps. Moreover, studies have shown variability in surgeons' performances, which is valuable to expose residents to different ways to perform a procedure. However, it is unclear how to include the sequence of steps in training programs.

METHODS

We conducted a systematic review, including studies reporting explicit teaching of a standard sequence of steps, where assessment considered adherence to a standard sequence, and where faculty or students at any level participated. We searched for articles on PubMed, EMBASE, CINAHL, Web of Science, and Google Scholar databases.

RESULTS

We selected nine articles that met the inclusion criteria. The main strategy to teach the sequence was to use videos to demonstrate the procedure. The simulation was the main strategy to assess the learning of the sequence of steps. Non-standardized scoring protocols and written tests with variable validity evidence were the instruments used to assess the learning, and were focused on adherence to a standard sequence and the omission of steps.

CONCLUSIONS

Teaching and learning assessment of a standard sequence of steps is scarcely reported in procedural skills training literature. More research is needed to evaluate whether the new strategies to teach and assess the order of steps work. We recommend the use of Surgical Process Models and Surgical Data Science to incorporate the sequence of steps when teaching and assessing procedural skills.

Surgical gestures as a method to quantify surgical performance and predict patient outcomes

How well a surgery is performed impacts a patient's outcomes; however, objective quantification of performance remains an unsolved challenge. Deconstructing a procedure into discrete instrument-tissue "gestures" is a emerging way to understand surgery. To establish this paradigm in a procedure where performance is the most important factor for patient outcomes, we identify 34,323 individual gestures performed in 80 nerve-sparing robot-assisted radical prostatectomies from two international medical centers. Gestures are classified into nine distinct dissection gestures (e.g., hot cut) and four supporting gestures (e.g., retraction). Our primary outcome is to identify factors impacting a patient's 1-year erectile function (EF) recovery after radical prostatectomy. We find that less use of hot cut and more use of peel/push are statistically associated with better chance of 1-year EF recovery. Our results also show interactions between surgeon experience and gesture types-similar gesture selection resulted in different EF recovery rates dependent on surgeon experience. To further validate this framework, two teams independently constructe distinct machine learning models using gesture sequences vs. traditional clinical features to predict 1-year EF. In both models, gesture sequences are able to better predict 1-year EF (Team 1: AUC 0.77, 95% CI 0.73-0.81; Team 2: AUC 0.68, 95% CI 0.66-0.70) than traditional clinical features (Team 1: AUC 0.69, 95% CI 0.65-0.73; Team 2: AUC 0.65, 95% CI 0.62-0.68). Our results suggest that gestures provide a granular method to objectively indicate surgical performance and outcomes. Application of this methodology to other surgeries may lead to discoveries on methods to improve surgery.

Validation study of a skill assessment tool for education and outcome prediction of laparoscopic distal gastrectomy

BACKGROUND

The Japanese operative-rating scale for laparoscopic distal gastrectomy (JORS-LDG) was developed through cognitive task analysis together with the Delphi method to measure intraoperative performance during laparoscopic distal gastrectomy. This study aimed to investigate the value of this rating scale as an educational tool and a surgical outcome predictor in laparoscopic distal gastrectomy.

METHODS

The surgical performance of laparoscopic distal gastrectomy was assessed by the first assistant, through self-evaluation in the operating room and by video raters blind to the case. We evaluated inter-rater reliability, internal consistency, and correlations between the JORS-LDG scores and the evaluation methods, patient characteristics, and surgical outcomes.

RESULTS

Fifty-four laparoscopic distal gastrectomy procedures performed by 40 surgeons at 16 institutions were evaluated in the operating room and with video recordings using the proposed rating scale. The video inter-rater reliability was > 0.8. Participating surgeons were divided into the low, intermediate, and high groups based on their total scores. The number of laparoscopic surgeries and laparoscopic gastrectomy procedures performed differed significantly among the groups according to laparoscopic distal gastrectomy skill levels. The low, intermediate, and high groups also differed in terms of median operating times (311, 266, and 229 min, respectively, P < 0.001), intraoperative complication rates (27.8, 11.8, and 0%, respectively, P = 0.01), and postoperative complication rates (22.2, 0, and 0%, respectively, P = 0.002).

CONCLUSIONS

The JORS-LDG is a reliable and valid measure for laparoscopic distal gastrectomy training and could be useful in predicting surgical outcomes.

The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review

BACKGROUND

Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes.

METHODS

Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis.

RESULTS

A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results.

CONCLUSION

Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.

Assessing the efficacy of dissection gestures in robotic surgery

Our group previously defined a dissection gesture classification system that deconstructs robotic tissue dissection into its most elemental yet meaningful movements. The purpose of this study was to expand upon this framework by adding an assessment of gesture efficacy (ineffective, effective, or erroneous) and analyze dissection patterns between groups of surgeons of varying experience. We defined three possible gesture efficacies as ineffective (no meaningful effect on the tissue), effective (intended effect on the tissue), and erroneous (unintended disruption of the tissue). Novices (0 prior robotic cases), intermediates (1-99 cases), and experts (≥ 100 cases) completed a robotic dissection task in a dry-lab training environment. Video recordings were reviewed to classify each gesture and determine its efficacy, then dissection patterns between groups were analyzed. 23 participants completed the task, with 9 novices, 8 intermediates with median caseload 60 (IQR 41-80), and 6 experts with median caseload 525 (IQR 413-900). For gesture selection, we found increasing experience associated with increasing proportion of overall dissection gestures (p = 0.009) and decreasing proportion of retraction gestures (p = 0.009). For gesture efficacy, novices performed the greatest proportion of ineffective gestures (9.8%, p < 0.001), intermediates commit the greatest proportion of erroneous gestures (26.8%, p < 0.001), and the three groups performed similar proportions of overall effective gestures, though experts performed the greatest proportion of effective retraction gestures (85.6%, p < 0.001). Between groups of experience, we found significant differences in gesture selection and gesture efficacy. These relationships may provide insight into further improving surgical training.

Machine Learning to Delineate Surgeon and Clinical Factors That Anticipate Positive Surgical Margins After Robot-Assisted Radical Prostatectomy

Purpose: Automated performance metrics (APMs), derived from instrument kinematic and systems events data during robotic surgery, are validated objective measures of surgeon performance. Our previous studies showed that APMs are strong outcome predictors of urinary continence after robot-assisted radical prostatectomy (RARP). We now use machine learning to investigate how surgeon performance (i.e., APMs) and clinical factors can predict positive surgical margins (PSMs) after RARP. Methods: We prospectively collected data of patients undergoing RARP at our institution from 2016 to 2019. Random Forest model predicted PSMs based on 15 clinical factors and 38 APMs from 11 standardized RARP steps. Out-of-bag Gini impurity index determined the top 10 variables of importance (VOI). APMs in the top 10 VOI were assessed for confounding effects by extracapsular extension (ECE) and pathologic T (pT) through Poisson regression with Generalized Estimating Equation. Results: 55/236 (23.3%) cases had PSMs. Of the 55 cases with PSMs, 9 (16.4%) were pT2 and 46 (83.6%), pT3. The full model, including clinical factors and APMs, achieved area under the curve (AUC) 0.74. When assessing clinical factors or APMs alone, the model achieved AUC 0.72 and 0.64, respectively. The strongest PSM predictors were ECE and pT stage, followed by APMs in specific steps. After adjusting for ECE and pT stage, most APMs remained as independent predictors of PSM. Conclusion: Using machine learning methods, we found that the strongest predictors of PSMs after RARP are nonmodifiable, disease-driven factors (ECE and pT). While APMs provide minimal additional insight into when PSMs may occur, they are nonetheless capable of independently predicting PSMs based on objective measures of surgeon performance.

Analyzing and Discussing Human Factors Affecting Surgical Patient Safety Using Innovative Technology: Creating a Safer Operating Culture

INTRODUCTION

Surgical errors often occur because of human factor-related issues. A medical data recorder (MDR) may be used to analyze human factors in the operating room. The aims of this study were to assess intraoperative safety threats and resilience support events by using an MDR and to identify frequently discussed safety and quality improvement issues during structured postoperative multidisciplinary debriefings using the MDR outcome report.

METHODS

In a cross-sectional study, 35 standard laparoscopic procedures were performed and recorded using the MDR. Outcome data were analyzed using the automated Systems Engineering Initiative for Patient Safety model. The video-assisted MDR outcome report reflects on safety threat and resilience support events (categories: person, tasks, tools and technology, psychical and external environment, and organization). Surgeries were debriefed by the entire team using this report. Qualitative data analysis was used to evaluate the debriefings.

RESULTS

A mean (SD) of 52.5 (15.0) relevant events were identified per surgery. Both resilience support and safety threat events were most often related to the interaction between persons (272 of 360 versus 279 of 400). During the debriefings, communication failures (also category person) were the main topic of discussion.

CONCLUSIONS

Patient safety threats identified by the MDR and discussed by the operating room team were most frequently related to communication, teamwork, and situational awareness. To create an even safer operating culture, educational and quality improvement initiatives should aim at training the entire operating team, as it contributes to a shared mental model of relevant safety issues.

Outcome prediction in bariatric surgery through video-based assessment

INTRODUCTION

The relationship between intraoperative surgical performance scores and patient outcomes has not been demonstrated at a single-case level. The GEARS score is a Likert-based scale that quantifies robotic surgical proficiency in 5 domains. Given that even highly skilled surgeons can have variability in their skill among their cases, we hypothesized that at a patient level, higher surgical skill as determined by the GEARS score will predict individual patient outcomes.

METHODS

Patients undergoing robotic sleeve gastrectomy between July 2018 and January 2021 at a single-health care system were captured in a prospective database. Bivariate Pearson's correlation was used to compare continuous variables, one-way ANOVA for categorical variables compared with a continuous variable, and chi-square for two categorical variables. Significant variables in the univariable screen were included in a multivariable linear regression model. Two-tailed p-value < 0.05 was considered significant.

RESULTS

Of 162 patients included, 9 patients (5.5%) experienced a serious morbidity within 30 days. The average excess weight loss (EWL) was 72 ± 12% at 6 months and 74 ± 15% at 12 months. GEARS score was not significantly correlated with EWL at 6 months (p = 0.349), 12 months (p = 0.468), or serious morbidity (p = 0.848) on unadjusted analysis. After adjusting, total GEARS score was not correlated with serious morbidity (p = 0.914); however, GEARS score did predict EWL at 6 (p < 0.001) and 12 months (p < 0.001). All GEARS subcomponent scores, bimanual dexterity, depth perception, efficiency, force sensitivity, and robotic control were predictive of EWL at 6 months (p < 0.001) and 12 months (p < 0.001) on multivariable analysis.

CONCLUSION

For patients undergoing sleeve gastrectomy, surgical skill as assessed by the GEARS score was correlated with EWL, suggesting that better performance of a sleeve gastrectomy can result in improved postoperative weight loss.

Standardization of ileoanal J-pouch surgery technique: Quality assessment of minimally invasive ileoanal J-pouch surgery videos

BACKGROUND

Ileal pouch anal anastomosis is a complex procedure associated with significant morbidity, with several complications after ileal pouch anal anastomosis surgery leading to pouch failure. The aim of the study is to evaluate the heterogeneity surrounding the technique of ileoanal J-pouch surgery by assessing the safety and quality of published online peer-reviewed surgical videos.

METHODS

Ileal pouch anal anastomosis videos published on peer-reviewed surgical journals and video channels were edited and anonymized to demonstrate specific steps of the surgical procedure: mobilization and division of the rectum, formation of the ileoanal J-pouch reservoir, anastomosis, and lengthening techniques. The anonymized videos were presented to a group of reviewers with expertise in ileal pouch anal anastomosis blinded to the names and affiliations of the surgeons performing the procedure. Primary outcome was the rate of interobserver variability in the assessment of specific technical steps of the ileal pouch anal anastomosis surgery procedure. Secondary outcome was the appropriateness of the use of surgical videos review as an assessment tool for ileal pouch anal anastomosis surgery, measured as rate of reviewers being unable to answer for poor video quality.

RESULTS

In total, 29 video fragments were distributed, and 13 assessors completed a 60-item survey, organized in 7 major domains. The survey completion rate was 93.4%. Out of a total 729 answers, in 23 (3.2%) the reviewers indicated they were unable to comment due to poor video image, and in 48 (6.5%) were unable to comment due to the particular step not being shown in the procedure. The proportion of assessors rating rectal mobilization technically appropriate ranged from 30.7% to 92.3% and from 7.7% to 69.2% for safety. The level of rectal division was considered appropriate in 0 to 53.8% of the videos, whereas the stapling technique used for rectal division was appropriate in 0 to 70% of the videos.

CONCLUSION

Our study assessed published peer-reviewed videos on ileal pouch anal anastomosis surgery and reported heterogeneity in the safety of the demonstrated techniques. Blind assessment of published peer-reviewed ileal pouch anal anastomosis videos reported a high rate of unsafe or inappropriate technique for rectal mobilization and transection in the reviewed videos, with fair interobserver agreement among reviewers. There is a need for consensus on what is considered safe and appropriate in ileal pouch anal anastomosis surgery. Peer review of ileal pouch anal anastomosis surgery videos could facilitate training and accreditation in this complex procedure.

Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial

IMPORTANCE

To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches.

OBJECTIVE

To determine how VOA and remote expert instruction compare in learners' skill acquisition, affective, and cognitive outcomes during surgical simulation training.

DESIGN, SETTING, AND PARTICIPANTS

This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021.

INTERVENTIONS

The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback.

MAIN OUTCOMES AND MEASURES

The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, -1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports.

RESULTS

A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group. All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001). Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups. However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found.

CONCLUSIONS AND RELEVANCE

In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT04700384.

Intraoperative dynamics of workflow disruptions and surgeons' technical performance failures: insights from a simulated operating room

INTRODUCTION

Flow disruptions (FD) in the operating room (OR) have been found to adversely affect the levels of stress and cognitive workload of the surgical team. It has been concluded that frequent disruptions also lead to impaired technical performance and subsequently pose a risk to patient safety. However, respective studies are scarce. We therefore aimed to determine if surgical performance failures increase after disruptive events during a complete surgical intervention.

METHODS

We set up a mixed-reality-based OR simulation study within a full-team scenario. Eleven orthopaedic surgeons performed a vertebroplasty procedure from incision to closure. Simulations were audio- and videotaped and key surgical instrument movements were automatically tracked to determine performance failures, i.e. injury of critical tissue. Flow disruptions were identified through retrospective video observation and evaluated according to duration, severity, source, and initiation. We applied a multilevel binary logistic regression model to determine the relationship between FDs and technical performance failures. For this purpose, we compared FDs in one-minute intervals before performance failures with intervals without subsequent performance failures.

RESULTS

Average simulation duration was 30:02 min (SD = 10:48 min). In 11 simulated cases, 114 flow disruption events were observed with a mean hourly rate of 20.4 (SD = 5.6) and substantial variation across FD sources. Overall, 53 performance failures were recorded. We observed no relationship between FDs and likelihood of immediate performance failures: Adjusted odds ratio = 1.03 (95% CI 0.46-2.30). Likewise, no evidence could be found for different source types of FDs.

CONCLUSION

Our study advances previous methodological approaches through the utilisation of a mixed-reality simulation environment, automated surgical performance assessments, and expert-rated observations of FD events. Our data do not support the common assumption that FDs adversely affect technical performance. Yet, future studies should focus on the determining factors, mechanisms, and dynamics underlying our findings.

Cognitive task analysis-based training in surgery: a meta-analysis

BACKGROUND

Reduced hands-on operating experience has challenged the development of complex decision-making skills for modern surgical trainees. Cognitive task analysis- (CTA-)based training is a methodical solution to extract the intricate cognitive processes of experts and impart this information to novices. Its use has been successful in high-risk industries such as the military and aviation, though its application for learning surgery is more recent. This systematic review aims to synthesize the evidence evaluating the efficacy of CTA-based training to enable surgeons to acquire procedural skills and knowledge.

METHODS

The PRISMA guidelines were followed. Four databases, including MEDLINE, EMBASE, Web of Science and Cochrane CENTRAL, were searched from inception to February 2021. Randomized controlled trials and observational studies evaluating the training effect of CTA-based interventions on novices' procedural knowledge or technical performance were included. Meta-analyses were performed using a random-effects model.

RESULTS

The initial search yielded 2205 articles, with 12 meeting the full inclusion criteria. Seven studies used surgical trainees as study subjects, four used medical students and one study used a combination. Surgical trainees enrolled into CTA-based training groups had enhanced procedural knowledge (standardized mean difference (SMD) 1.36 (95 per cent c.i. 0.67 to 2.05), P < 0.001) and superior technical performance (SMD 2.06 (95 per cent c.i. 1.17 to 2.96), P < 0.001) in comparison with groups that used conventional training methods.

CONCLUSION

CTA-based training is an effective way to learn the cognitive skills of a surgical procedure, making it a useful adjunct to current surgical training.

Variation in Surgeon Proficiency Scores and Association With Digit Replantation Outcomes

IMPORTANCE

Risk-adjusted variation in surgeon outcomes has been traditionally explained by surgeon volume and hospital infrastructure, yet it is unclear how a surgeon's operative proficiency directly contributes to their patients' outcomes.

OBJECTIVE

To assess the variation of surgeons' operative proficiency and investigate its association with surgical outcomes.

DESIGN, SETTING, AND PARTICIPANTS

This case series was a retrospective analysis of all digit replantations and revascularizations at a single US university medical center between January 2000 and August 2020. Surgeons were assigned a proficiency score based on the expected procedure difficulty and outcomes from a sample of their cases. Surgeon proficiency scores were then used to determine associations with outcomes from subsequent cases. The expected difficulty of each case was calculated using a novel scoring system that applied pooled relative risks from a meta-analysis of risk factors for replantation and revascularization failure.

EXPOSURES

Digit replantation and revascularization.

MAIN OUTCOMES AND MEASURES

Digit survival at 1-month follow up (case success) and number of complications.

RESULTS

A total of 145 patients and 226 digits were treated by 11 surgeons with training in hand or microsurgery (mean [SD] age, 41.9 [15.2] years; 204 [90%] men); there were 116 replantations and 110 revascularizations. Surgeon proficiency scores ranged from 1.3 to 5.7, with a mean (SD) of 3.4 (1.4). Case success rates among surgeons varied from 20.0% to 90.5%, with a mean (SD) of 64.9%. Higher proficiency scores were associated with fewer case failures: each point increase was associated with 40% decreased odds of failure (odds ratio, 0.60; 95% CI, 0.38-0.94). Every 3-point increase in proficiency score was associated with 1 less complication (effect estimate, -0.29; 95% CI, -0.56 to 0.02). Surgeon proficiency score had a greater association with case failure than surgeon volume (16.7% vs 12.0%). The final model's association with case failure had an area under the receiver operating characteristics curve of 0.93.

CONCLUSIONS AND RELEVANCE

Operative proficiency varied widely among practicing surgeons and accounted for 17% of estimative ability for success of digit replantation and revascularization. Greater surgeon proficiency was associated with better outcomes, indicating that the value of surgical care may be optimized by improving surgeon proficiency.

Effect assessment of the application value of evidence-based nursing intervention in operating room nursing: A protocol for a systematic review and meta-analysis

BACKGROUND

The advantages of evidence-based nursing (EBN) intervention in health care settings have been widely disseminated to nurses throughout the world. More researches are reporting the effectiveness of EBN intervention in operating room nursing. However, the results are inconsistent. This study focuses on conducting a meta-analysis and systematic evaluation aimed at determining the usefulness of EBN intervention in operating room nursing.

METHODS

The Preferred Reporting Items for Systematic Reviews and Meta-analysis for Protocols criteria were used to write this paper. We will look for relevant studies from 2 Chinese databases (China National Knowledge Infrastructure and Wanfang database and also from 3 English databases such as Web of Science, Cochrane Library, PubMed, and EMBASE), to locate all relevant randomized controlled trials and observational studies assessing the application value of EBN intervention in operating room nursing from their commencement to June 2021. Separately, 2 authors will choose the studies, do the data extract and conduct the assessment probing into the likelihood of bias. If there is a disagreement, it will be resolved by the third author. RevMan 5.3 software and Stata 15.0 software will be used to conduct the meta-analysis.

RESULTS

The usefulness of EBN intervention in operating room nursing will be assessed in this study.

CONCLUSION

The purpose of this research is to conclude the value of EBN intervention in operating room nursing and the quality of current data.

ETHICS AND DISSEMINATION

Since there is no requirement for data on the individual patient, hence there will be no need for ethical approval.

OSF NUMBER

DOI 10.17605/OSF.IO/MSXNF.

Tools for measuring technical skills during gynaecologic surgery: a scoping review

BACKGROUND

Standardised assessment is key to structured surgical training. Currently, there is no consensus on which surgical assessment tool to use in live gynaecologic surgery. The purpose of this review is to identify assessment tools measuring technical skills in gynaecologic surgery and evaluate the measurement characteristics of each tool.

METHOD

We utilized the scoping review methodology and searched PubMed, Medline, Embase and Cochrane. Inclusion criteria were studies that analysed assessment tools in live gynaecologic surgery. Kane's validity argument was applied to evaluate the assessment tools in the included studies.

RESULTS

Eight studies out of the 544 identified fulfilled the inclusion criteria. The assessment tools were categorised as global rating scales, global and procedure rating scales combined, procedure-specific rating scales or as a non-procedure-specific error assessment tool.

CONCLUSION

This scoping review presents the current different tools for observational assessment of technical skills in intraoperative, gynaecologic surgery. This scoping review can serve as a guide for surgical educators who want to apply a scale or a specific tool in surgical assessment.

Association Between Surgeon Technical Skills and Patient Outcomes

Importance

Postoperative complications remain common after surgery, but little is known about the extent of variation in operative technical skill and whether variation is associated with patient outcomes.

Objectives

To examine the (1) variation in technical skill scores of practicing surgeons, (2) association between technical skills and patient outcomes, and (3) amount of variation in patient outcomes explained by a surgeon's technical skill.

Design, Setting, and Participants

In this quality improvement study, 17 practicing surgeons submitted a video of a laparoscopic right hemicolectomy that was then rated by at least 10 blinded peer surgeons and 2 expert raters. The association between surgeon technical skill scores and risk-adjusted outcomes was examined using data from the American College of Surgeons National Surgical Quality Improvement Program. The association between technical skill scores and outcomes was examined for colorectal procedures and noncolorectal procedures (ie, assessed on whether technical skills demonstrated during colectomy were associated with patient outcomes across other cases). In addition, the proportion of patient outcomes explained by technical skill scores was examined using robust regression techniques. The study was conducted from September 23, 2016, to February 10, 2018; data analysis was performed from November 2018 to January 2019.

Exposures

Colorectal and noncolorectal procedures.

Main Outcomes and Measures

Any complication, mortality, unplanned hospital readmission, unplanned reoperation related to principal procedure, surgical site infection, and death or serious morbidity.

Results

Of the 17 surgeons included in the study, 13 were men (76%). The participants had a range from 1 to 28 years in surgical practice (median, 11 years). Based on 10 or more reviewers per video and with a maximum quality score of 5, overall technical skill scores ranged from 2.8 to 4.6. From 2014 to 2016, study participants performed a total of 3063 procedures (1120 colectomies). Higher technical skill scores were significantly associated with lower rates of any complication (15.5% vs 20.6%, P = .03; Spearman rank-order correlation coefficient r = -0.54, P = .03), unplanned reoperation (4.7% vs 7.2%, P = .02; r = -0.60, P = .01), and a composite measure of death or serious morbidity (15.9% vs 21.4%, P = .02; r = -0.60, P = .01) following colectomy. Similar associations were found between colectomy technical skill scores and patient outcomes for all types of procedures performed by a surgeon. Overall, technical skill scores appeared to account for 25.8% of the variation in postcolectomy complication rates and 27.5% of the variation when including noncolectomy complication rates.

Conclusions and Relevance

The findings of this study suggest that there is wide variation in technical skill among practicing surgeons, accounting for more than 25% of the variation in patient outcomes. Higher colectomy technical skill scores appear to be associated with lower complication rates for colectomy and for all other procedures performed by a surgeon. Efforts to improve surgeon technical skills may result in better patient outcomes.

Objective evaluation of technical dexterity in robotic hepaticojejunostomy: Assessment of hepatopancreatobiliary fellows using cumulative sum analytics

BACKGROUND

The development of technical dexterity is a critical for surgeons in training. This study describes and assesses the feasibility of an objective method for the evaluation of procedure-specific technical dexterity in hepatopancreatobiliary (HPB) surgery using cumulative sum (CUSUM) analysis.

METHODS

Dry-lab HPB procedures were divided into procedural steps with binary outcomes (success or failure). Two HPB fellows completed 20 dry lab hepaticojejunostomy (HJ) procedures. Participant progress was tracked over time with CUSUM analytics to establish a learning curve for procedural proficiency.

RESULTS

The CUSUM charts for 20 consecutive dry-lab HJ procedures were analysed. A learning curve was created and used to identify areas of weakness to facilitate improvement in technical proficiency.

CONCLUSIONS

CUSUM is effective tool for objective evaluation of technical dexterity offering both simplicity and adaptability. We demonstrate its use and feasibility for surgical education and plan to expand its' application to assess residents performing general surgery procedures.

Development and clinical implementation of a structured, simulation-based training programme in laparoscopic appendectomy: description, validation and evaluation

Background

Laparoscopic appendectomy is a common procedure in general surgery but is likely underused in structured and real-life teaching. This study describes the development, validation and evaluation of implementing a structured training programme for laparoscopic appendectomy.

Study design

A structured curriculum and simulation-based programme for trainees and trainers was developed. All general surgery trainees and trainers were involved in laparoscopic appendectomies. All trainees and trainers underwent the structured preprocedure training programme before real-life surgery evaluation. A standardised form evaluated eight technical steps (skills) of the procedure as well as an overall assessment, and nine elements of communication (feedback), and was used for bilateral evaluation by each trainee and trainer. A consecutive, observational cohort over a 12-month period was used to gauge real-life implementation.

Results

During 277 eligible real-life appendectomies, structured evaluation was performed in 173 (62%) laparoscopic appendectomies, for which 165 forms were completed by 19 trainees. Construct validity was found satisfactory. Inter-rater reliability demonstrated good correlation between trainee and trainer. The trainees’ and trainers’ stepwise and overall assessments of technical skills had an overall good reliability (intraclass correlation coefficient of 0.88). The vast majority (92.2%) of the trainees either agreed or strongly agreed that the training met their expectations.

Conclusion

Structured training for general surgery residents can be implemented for laparoscopic appendectomy. Skills assessment by trainees and trainers indicated reliable self-assessment. Overall, the trainees were satisfied with the training, including the feedback from the trainers.

Decision points in pancreatoduodenectomy: Insights from the contemporary experts on prevention, mitigation, and management of postoperative pancreatic fistula

BACKGROUND

Despite abundant, high-level scientific evidence, there is no consensus regarding the prevention, mitigation, and management of clinically relevant pancreatic fistula after pancreatoduodenectomy. The aim of the present investigation is three-fold: (1) to analyze the multiple decision-making points for pancreatico-enteric anastomotic creation and fistula mitigation and management after pancreatoduodenectomy, (2) to reveal the practice of contemporary experts, and (3) to indicate avenues for future research to reduce the burden of clinically relevant pancreatic fistula.

METHODS

A 109-item questionnaire was sent to a panel of international pancreatic surgery experts, recognized for their clinical and scientific authority. Their practice habits and thought processes regarding clinically relevant pancreatic fistula risk assessment, anastomotic construction, application of technical adjuncts, and mitigation strategies, as well as postoperative management, was explored. Sixteen clinical vignettes were presented to reveal their certain approaches to unique situations-both common and uncommon.

RESULTS

Sixty experts, with a cumulative 48,860 pancreatoduodenectomies, completed the questionnaire. Their median pancreatectomy/pancreatoduodenectomy case volume was 1,200 and 705 procedures, respectively, with a median career duration of 22 years and 200 indexed publications. Although pancreatico-jejunostomy reconstruction with transperitoneal drainage is the standard operative approach for most authorities, uncertainty emerges regarding the employment of objective risk stratification and adaptation of practice to risk. Concrete suggestions are offered to inform decision-making in intimidating circumstances. Early drain removal is frequently embraced, while a step-up approach is unanimously invoked to treat severe clinically relevant pancreatic fistula.

CONCLUSION

A comprehensive conceptual framework of 4 sequential phases of decision-making is proposed-risk assessment, anastomotic technique, mitigation strategy employment, and postoperative management. Basic science studies and outcome analyses are proposed for improvement.

Clinical Evaluation of Intraoperative Near Misses in Laparoscopic Rectal Cancer Surgery

OBJECTIVE

To investigate the frequency, nature, and severity of intraoperative adverse near miss events within advanced laparoscopic surgery and report any associated clinical impact.

BACKGROUND

Despite implementation of surgical safety initiatives, the intraoperative period is poorly documented with evidence of underreporting. Near miss analyses are undertaken in high-risk industries but not in surgical practice.

METHODS

Case video and data from 2 laparoscopic total mesorectal excision randomized controlled trials were analyzed (ALaCaRT ACTRN12609000663257, 2D3D ISRCTN59485808). Intraoperative adverse events were identified and categorized using the observational clinical human reliability analysis technique. The EAES classification was applied by 2 blinded assessors. EAES grade 1 events (nonconsequential error, no damage, or need for correction) were considered near misses. Associated clinical impact was assessed with early morbidity and histopathology outcomes.

RESULTS

One hundred seventy-five cases contained 1113 error events. Six hundred ninety-eight (62.7%) were near misses (median 3, IQR 2-5, range 0-15) with excellent inter-rater and test-retest reliability (κ=0.86, 95% CI 0.83-0.89, P < 0.001 and κ=0.88, 95% CI 0.85-0.9, P < 0.001 respectively). Significantly more near misses were seen in patients who developed early complications (4 (3-6) vs. 3 (2-4), P < 0.001). Higher numbers of near misses were seen in patients with more numerous (P = 0.002) and more serious early complications (P = 0.003). Cases containing major intraoperative adverse events contained significantly more near misses (5 (3-7) vs. 3 (2-5), P < 0.001) with a major event observed for every 19.4 near misses.

CONCLUSION

Intraoperative adverse events and near misses can be reliably and objectively captured in advanced laparoscopic surgery. Near misses are commonplace and closely associated with morbidity outcomes.

Influencing factors and strategy to the career planning of operating room nurses

Aim

This study aims to investigate the status quo of self‐career planning of nurses working in operating rooms (OR) and analyse the influencing factors.

Methods

A total of 1,418 nurses working in OR were selected and surveyed using a general data questionnaire, a nurse career planning questionnaire and an organizational career management questionnaire.

Results

Multiple linear regression analysis showed that organizational career management perception, hospital‐grade and monthly income were the influencing factors of self‐career planning score of nurses working in OR, which could explain the 56.6% variation of career organization management.

Conclusion

The status quo of self‐career planning in OR nurse was at the middle level, and the scores of different dimensions were not balanced. Reasonable self‐career planning helps the OR nurses improve their professional skills and increase their commitment to work and satisfaction.

Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery

Importance

Complex surgical interventions are inherently prone to variation yet they are not objectively measured. The reasons for outcome differences following cancer surgery are unclear.

Objective

To quantify surgical skill within advanced laparoscopic procedures and its association with histopathological and clinical outcomes.

Design, Setting, and Participants

This analysis of data and video from the Australasian Laparoscopic Cancer of Rectum (ALaCaRT) and 2-dimensional/3-dimensional (2D3D) multicenter randomized laparoscopic total mesorectal excision trials, which were conducted at 28 centers in Australia, the United Kingdom, and New Zealand, was performed from 2018 to 2019 and included 176 patients with clinical T1 to T3 rectal adenocarcinoma 15 cm or less from the anal verge. Case videos underwent blinded objective analysis using a bespoke performance assessment tool developed with a 62-international expert Delphi exercise and workshop, interview, and pilot phases.

Interventions

Laparoscopic total mesorectal excision undertaken with curative intent by 34 credentialed surgeons.

Main Outcomes and Measures

Histopathological (plane of mesorectal dissection, ALaCaRT composite end point success [mesorectal fascial plane, circumferential margin, ≥1 mm; distal margin, ≥1 mm]) and 30-day morbidity. End points were analyzed using surgeon quartiles defined by tool scores.

Results

The laparoscopic total mesorectal excision performance tool was produced and shown to be reliable and valid for the specialist level (intraclass correlation coefficient, 0.889; 95% CI, 0.832-0.926; P < .001). A substantial variation in tool scores was recorded (range, 25-48). Scores were associated with the number of intraoperative errors, plane of mesorectal dissection, and short-term patient morbidity, including the number and severity of complications. Upper quartile-scoring surgeons obtained excellent results compared with the lower quartile (mesorectal fascial plane: 93% vs 59%; number needed to treat [NNT], 2.9, P = .002; ALaCaRT end point success, 83% vs 58%; NNT, 4; P = .03; 30-day morbidity, 23% vs 50%; NNT, 3.7; P = .03).

Conclusions and Relevance

Intraoperative surgical skill can be objectively and reliably measured in complex cancer interventions. Substantial variation in technical performance among credentialed surgeons is seen and significantly associated with clinical and pathological outcomes.

Quantifying the "Assistant Effect" in Robotic-Assisted Radical Prostatectomy (RARP): Measures of Technical Performance

PURPOSE

Surgeons are reliant on the bedside assistant during robotic surgeries. Using a modified global rating scale (GRS), we aim to assess the association between an assistant's technical skill on surgeon performance in Robotic-Assisted Radical Prostatectomy (RARP).

METHODS

Prospective, intraoperative video from RARP cases at three centers were collected. Baseline demographic and RARP-experience data were collected from participating surgeons and trainees. The dissection of the prostatic pedicle and neurovascular bundle step (NVB) was analyzed. Expert analysts scored the console surgeon performance using the Global Evaluative Assessment of Robotic Skills (GEARS), and the bedside assistant performance using a modified Objective Structured Assessment of Technical Skills (aOSATS). The primary outcome is the association between console surgeon performance, as measured by GEARS, and assistant skill, as measured by aOSATS. Spearman's rho correlations were used to test the relationship between assistant and surgeon technical performance, and a multivariable linear regression model was created to test this association while controlling for patient factors.

RESULTS

92 RARP cases were available for the analysis, comprising 14 console surgeons and 22 different bedside assistants. In only 5 (5.4%) cases, the neurovascular bundle step was completed by a trainee, and in 13 (14.1%) of cases, a staff-level surgeon acted as the bedside assistant. aOSATS score was significantly associated with robotic console experience (P = 0.011), and prior laparoscopic experience (P < 0.001). Assistant aOSATS score showed a weak but significant correlation with surgeon GEARS score during the neurovascular bundle step (spearman's rho = 0.248, P = 0.028). On linear regression, aOSATS remained a significant predictor of console surgeon performance (P = 0.016), after controlling for patient age and BMI, prostate volume, tumor stage, and presence of nerve-sparing.

CONCLUSIONS

This is the first study to assess the association between assistant technical skill and surgeon performance in RARP. Additionally, we have provided validity evidence for a modified OSATS global rating scale for training and assessing bedside assistant performance.

Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience

Automated performance metrics objectively measure surgeon performance during a robot-assisted radical prostatectomy. Machine learning has demonstrated that automated performance metrics, especially during the vesico-urethral anastomosis of the robot-assisted radical prostatectomy, are predictive of long-term outcomes such as continence recovery time. This study focuses on automated performance metrics during the vesico-urethral anastomosis, specifically on stitch versus sub-stitch levels, to distinguish surgeon experience. During the vesico-urethral anastomosis, automated performance metrics, recorded by a systems data recorder (Intuitive Surgical, Sunnyvale, CA, USA), were reported for each overall stitch (C^total) and its individual components: needle handling/targeting (C¹), needle driving (C²), and suture cinching (C³) (Fig 1, A). These metrics were organized into three datasets (GlobalSet [whole stitch], RowSet [independent sub-stitches], and ColumnSet [associated sub-stitches] (Fig 1, B) and applied to three machine learning models (AdaBoost, gradient boosting, and random forest) to solve two classifications tasks: experts (≥100 cases) versus novices (<100 cases) and ordinary experts (≥100 and <2,000 cases) versus super experts (≥2,000 cases). Classification accuracy was determined using analysis of variance. Input features were evaluated through a Jaccard index. From 68 vesico-urethral anastomoses, we analyzed 1,570 stitches broken down into 4,708 sub-stitches. For both classification tasks, ColumnSet best distinguished experts (n = 8) versus novices (n = 9) and ordinary experts (n = 5) versus super experts (n = 3) at an accuracy of 0.774 and 0.844, respectively. Feature ranking highlighted Endowrist articulation and needle handling/targeting as most important in classification. Surgeon performance measured by automated performance metrics on a granular sub-stitch level more accurately distinguishes expertise when compared with summary automated performance metrics over whole stitches.

Educational application of intraoperative records from an energy device in laparoscopic gastrectomy: a preliminary report

PURPOSE

Video review is a reliable method for surgical education in laparoscopic gastrectomy (LG), but more objective methods are still needed. The purpose of this study was to determine whether the energy device records reflected surgical competency, and thereby may improve surgical education.

METHODS

A total of 16 patients who underwent LG for gastric cancer using the Thunderbeat^® device were preliminarily retrospectively analyzed. This device has the function of 'intelligent tissue monitoring' (ITM), a safety assist system stopping energy output, and can record ITM detections and firing time during surgery. The number of ITM detections and firings, and the total firing time during gastrocolic ligament dissection and infrapyloric dissection were compared between trainees (n = 9 by 5 surgeons) and experts (n = 7 by 5 surgeons). The non-edited videos (n = 16) were scored, and the correlations between the scores and the records were then analyzed.

RESULTS

Significantly more ITM detections, firings, and a longer total firing time were observed in trainees than in experts. The number of ITM detections and firing had negative correlations with the scores of the operation speed, the use of the non-dominant hand, and the use of an energy device.

CONCLUSIONS

Our preliminary study suggested that the above described energy device records reflected surgical competency, and thereby may improve surgical education.

"GRASP" Module of Self-assessment with Virtual Mentoring for Uninterrupted Surgical Training During COVID-19 Pandemic

Disruption in surgical training during the COVID-19 pandemic has prompted many innovative "virtual" modules to address the loss of learning exposure during these difficult times. We suggest "GRASP" (Gain, Recognize, Analyze, Simulate, and Perform) module of self-assessment with virtual mentoring for uninterrupted surgical training. This idea merges the advantages of self-assessment and mentoring for the benefit of surgical trainees. Its embedded continuous close individualized mentoring can change the surgical training culture by initiating an assessment of surgical learning and skills right from the beginning of surgical training.

Delphi Method to Achieve Clinical Consensus for a BPMN Representation of the Central Venous Access Placement for Training Purposes

Proper teaching of the technical skills necessary to perform a medical procedure begins with its breakdown into its constituent steps. Currently available methodologies require substantial resources and their results may be biased. Therefore, it is difficult to generate the necessary breakdown capable of supporting a procedural curriculum. The aim of our work was to breakdown the steps required for ultrasound guided Central Venous Catheter (CVC) placement and represent this procedure graphically using the standard BPMN notation. Methods: We performed the first breakdown based on the activities defined in validated evaluation checklists, which were then graphically represented in BPMN. In order to establish clinical consensus, we used the Delphi method by conducting an online survey in which experts were asked to score the suitability of the proposed activities and eventually propose new activities. Results: Surveys were answered by 13 experts from three medical specialties and eight different institutions in two rounds. The final model included 28 activities proposed in the initial model and four new activities proposed by the experts; seven activities from the initial model were excluded. Conclusions: The proposed methodology proved to be simple and effective, generating a graphic representation to represent activities, decision points, and alternative paths. This approach is complementary to more classical representations for the development of a solid knowledge base that allows the standardization of medical procedures for teaching purposes.

Process-Oriented Instrument and Taxonomy for Teaching Surgical Procedures in Medical Training: The Ultrasound-Guided Insertion of Central Venous Catheter

Procedural training is relevant for physicians who perform surgical procedures. In the medical education field, instructors who teach surgical procedures need to understand how their students are learning to give them feedback and assess them objectively. The sequence of steps of surgical procedures is an aspect rarely considered in medical education, and state-of-the-art tools for giving feedback and assessing students do not focus on this perspective. Process Mining can help to include this perspective in this field since it has recently been used successfully in some applications. However, these previous developments are more centred on students than on instructors. This paper presents the use of Process Mining to fill this gap, generating a taxonomy of activities and a process-oriented instrument. We evaluated both tools with instructors who teach central venous catheter insertion. The results show that the instructors found both tools useful to provide objective feedback and objective assessment. We concluded that the instructors understood the information provided by the instrument since it provides helpful information to understand students’ performance regarding the sequence of steps followed.

Comparison of Nursing Performance Competencies and Practical Education Needs Based on Clinical Careers of Operating Room Nurses: A Cross-Sectional Study

In order to provide appropriate practical training for quality nursing care, it is necessary to evaluate nursing performance competencies in terms of clinical careers. Using convenience sampling, this cross-sectional study aimed to identify the nursing performance competencies and practical education needs of operating room nurses by evaluating nurses' clinical careers. The participants in this study were 182 nurses working in the operating rooms of four general hospitals in Seoul, Korea. Data came from self-report questionnaires regarding nursing performance competencies and practical education needs. The results showed that participants' nursing performance competencies and practical education needs differed significantly across clinical career groups. Further, participants' nursing performance competencies were negatively correlated with practical education needs in terms of clinical career. In order to develop an effective and continuous practical education program for operating nurses, various education programs that reflect nursing performance competencies and practical education needs for particular stages of clinical careers are necessary.

Towards near real-time assessment of surgical skills: A comparison of feature extraction techniques

BACKGROUND AND OBJECTIVE

Surgical skill assessment aims to objectively evaluate and provide constructive feedback for trainee surgeons. Conventional methods require direct observation with assessment from surgical experts which are both unscalable and subjective. The recent involvement of surgical robotic systems in the operating room has facilitated the ability of automated evaluation of the expertise level of trainees for certain representative maneuvers by using machine learning for motion analysis. The features extraction technique plays a critical role in such an automated surgical skill assessment system.

METHODS

We present a direct comparison of nine well-known feature extraction techniques which are statistical features, principal component analysis, discrete Fourier/Cosine transform, codebook, deep learning models and auto-encoder for automated surgical skills evaluation. Towards near real-time evaluation, we also investigate the effect of time interval on the classification accuracy and efficiency.

RESULTS

We validate the study on the benchmark robotic surgical training JIGSAWS dataset. An accuracy of 95.63, 90.17 and 90.26% by the Principal Component Analysis and 96.84, 92.75 and 95.36% by the deep Convolutional Neural Network for suturing, knot tying and needle passing, respectively, highlighted the effectiveness of these two techniques in extracting the most discriminative features among different surgical skill levels.

CONCLUSIONS

This study contributes toward the development of an online automated and efficient surgical skills assessment technique.