Surgical process modeling

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

BACKGROUND

Laparoscopic cholecystectomy is a very frequent surgical procedure. However, in an ageing society, less surgical staff will need to perform surgery on patients. Collaborative surgical robots (cobots) could address surgical staff shortages and workload. To achieve context-awareness for surgeon-robot collaboration, the intraoperative action workflow recognition is a key challenge.

METHODS

A surgical process model was developed for intraoperative surgical activities including actor, instrument, action and target in laparoscopic cholecystectomy (excluding camera guidance). These activities, as well as instrument presence and surgical phases were annotated in videos of laparoscopic cholecystectomy performed on human patients (n = 10) and on explanted porcine livers (n = 10). The machine learning algorithm Distilled-Swin was trained on our own annotated dataset and the CholecT45 dataset. The validation of the model was conducted using a fivefold cross-validation approach.

RESULTS

In total, 22,351 activities were annotated with a cumulative duration of 24.9 h of video segments. The machine learning algorithm trained and validated on our own dataset scored a mean average precision (mAP) of 25.7% and a top K = 5 accuracy of 85.3%. With training and validation on our dataset and CholecT45, the algorithm scored a mAP of 37.9%.

CONCLUSIONS

An activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities. The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant's tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.

Autonomous System for Tumor Resection (ASTR) - Dual-Arm Robotic Midline Partial Glossectomy

Head and neck cancers are the seventh most common cancers worldwide, with squamous cell carcinoma being the most prevalent histologic subtype. Surgical resection is a primary treatment modality for many patients with head and neck squamous cell carcinoma, and accurately identifying tumor boundaries and ensuring sufficient resection margins are critical for optimizing oncologic outcomes. This study presents an innovative autonomous system for tumor resection (ASTR) and conducts a feasibility study by performing supervised autonomous midline partial glossectomy for pseudotumor with millimeter accuracy. The proposed ASTR system consists of a dual-camera vision system, an electrosurgical instrument, a newly developed vacuum grasping instrument, two 6-DOF manipulators, and a novel autonomous control system. The letter introduces an ontology-based research framework for creating and implementing a complex autonomous surgical workflow, using the glossectomy as a case study. Porcine tongue tissues are used in this study, and marked using color inks and near-infrared fluorescent (NIRF) markers to indicate the pseudotumor. ASTR actively monitors the NIRF markers and gathers spatial and color data from the samples, enabling planning and execution of robot trajectories in accordance with the proposed glossectomy workflow. The system successfully performs six consecutive supervised autonomous pseudotumor resections on porcine specimens. The average surface and depth resection errors measure 0.73±0.60 mm and 1.89±0.54 mm, respectively, with no positive tumor margins detected in any of the six resections. The resection accuracy is demonstrated to be on par with manual pseudotumor glossectomy performed by an experienced otolaryngologist.

Surgical phase and instrument recognition: how to identify appropriate dataset splits

PURPOSE

Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split.

METHODS

We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits.

RESULTS

We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks.

CONCLUSION

In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/ .

A comparison between a maximum care university hospital and an outpatient clinic - potential for optimization in arthroscopic workflows?

BACKGROUND

Due to the growing economic pressure, there is an increasing interest in the optimization of operational processes within surgical operating rooms (ORs). Surgical departments are frequently dealing with limited resources, complex processes with unexpected events as well as constantly changing conditions. In order to use available resources efficiently, existing workflows and processes have to be analyzed and optimized continuously. Structural and procedural changes without prior data-driven analyses may impair the performance of the OR team and the overall efficiency of the department. The aim of this study is to develop an adaptable software toolset for surgical workflow analysis and perioperative process optimization in arthroscopic surgery.

METHODS

In this study, the perioperative processes of arthroscopic interventions have been recorded and analyzed subsequently. A total of 53 arthroscopic operations were recorded at a maximum care university hospital (UH) and 66 arthroscopic operations were acquired at a special outpatient clinic (OC). The recording includes regular perioperative processes (i.a. patient positioning, skin incision, application of wound dressing) and disruptive influences on these processes (e.g. telephone calls, missing or defective instruments, etc.). For this purpose, a software tool was developed ('s.w.an Suite Arthroscopic toolset'). Based on the data obtained, the processes of the maximum care provider and the special outpatient clinic have been analyzed in terms of performance measures (e.g. Closure-To-Incision-Time), efficiency (e.g. activity duration, OR resource utilization) as well as intra-process disturbances and then compared to one another.

RESULTS

Despite many similar processes, the results revealed considerable differences in performance indices. The OC required significantly less time than UH for surgical preoperative (UH: 30:47 min, OC: 26:01 min) and postoperative phase (UH: 15:04 min, OC: 9:56 min) as well as changeover time (UH: 32:33 min, OC: 6:02 min). In addition, these phases result in the Closure-to-Incision-Time, which lasted longer at the UH (UH: 80:01 min, OC: 41:12 min).

CONCLUSION

The perioperative process organization, team collaboration, and the avoidance of disruptive factors had a considerable influence on the progress of the surgeries. Furthermore, differences in terms of staffing and spatial capacities could be identified. Based on the acquired process data (such as the duration for different surgical steps or the number of interfering events) and the comparison of different arthroscopic departments, approaches for perioperative process optimization to decrease the time of work steps and reduce disruptive influences were identified.

LAST: LAtent Space-Constrained Transformers for Automatic Surgical Phase Recognition and Tool Presence Detection

When developing context-aware systems, automatic surgical phase recognition and tool presence detection are two essential tasks. There exist previous attempts to develop methods for both tasks but majority of the existing methods utilize a frame-level loss function (e.g., cross-entropy) which does not fully leverage the underlying semantic structure of a surgery, leading to sub-optimal results. In this paper, we propose multi-task learning-based, LAtent Space-constrained Transformers, referred as LAST, for automatic surgical phase recognition and tool presence detection. Our design features a two-branch transformer architecture with a novel and generic way to leverage video-level semantic information during network training. This is done by learning a non-linear compact presentation of the underlying semantic structure information of surgical videos through a transformer variational autoencoder (VAE) and by encouraging models to follow the learned statistical distributions. In other words, LAST is of structure-aware and favors predictions that lie on the extracted low dimensional data manifold. Validated on two public datasets of the cholecystectomy surgery, i.e., the Cholec80 dataset and the M2cai16 dataset, our method achieves better results than other state-of-the-art methods. Specifically, on the Cholec80 dataset, our method achieves an average accuracy of 93.12±4.71%, an average precision of 89.25±5.49%, an average recall of 90.10±5.45% and an average Jaccard of 81.11 ±7.62% for phase recognition, and an average mAP of 95.15±3.87% for tool presence detection. Similar superior performance is also observed when LAST is applied to the M2cai16 dataset.

Laparoscopic Video Analysis Using Temporal, Attention, and Multi-Feature Fusion Based-Approaches

Adapting intelligent context-aware systems (CAS) to future operating rooms (OR) aims to improve situational awareness and provide surgical decision support systems to medical teams. CAS analyzes data streams from available devices during surgery and communicates real-time knowledge to clinicians. Indeed, recent advances in computer vision and machine learning, particularly deep learning, paved the way for extensive research to develop CAS. In this work, a deep learning approach for analyzing laparoscopic videos for surgical phase recognition, tool classification, and weakly-supervised tool localization in laparoscopic videos was proposed. The ResNet-50 convolutional neural network (CNN) architecture was adapted by adding attention modules and fusing features from multiple stages to generate better-focused, generalized, and well-representative features. Then, a multi-map convolutional layer followed by tool-wise and spatial pooling operations was utilized to perform tool localization and generate tool presence confidences. Finally, the long short-term memory (LSTM) network was employed to model temporal information and perform tool classification and phase recognition. The proposed approach was evaluated on the Cholec80 dataset. The experimental results (i.e., 88.5% and 89.0% mean precision and recall for phase recognition, respectively, 95.6% mean average precision for tool presence detection, and a 70.1% F1-score for tool localization) demonstrated the ability of the model to learn discriminative features for all tasks. The performances revealed the importance of integrating attention modules and multi-stage feature fusion for more robust and precise detection of surgical phases and tools.

Operating Room Performance Optimization Metrics: a Systematic Review

Literature proposes numerous initiatives for optimization of the Operating Room (OR). Despite multiple suggested strategies for the optimization of workflow on the OR, its patients and (medical) staff, no uniform description of 'optimization' has been adopted. This makes it difficult to evaluate the proposed optimization strategies. In particular, the metrics used to quantify OR performance are diverse so that assessing the impact of suggested approaches is complex or even impossible. To secure a higher implementation success rate of optimisation strategies in practice we believe OR optimisation and its quantification should be further investigated. We aim to provide an inventory of the metrics and methods used to optimise the OR by the means of a structured literature study. We observe that several aspects of OR performance are unaddressed in literature, and no studies account for possible interactions between metrics of quality and efficiency. We conclude that a systems approach is needed to align metrics across different elements of OR performance, and that the wellbeing of healthcare professionals is underrepresented in current optimisation approaches.

The Sequence of Steps: A Key Concept Missing in Surgical Training-A Systematic Review and Recommendations to Include It

BACKGROUND

Surgical procedures have an inherent feature, which is the sequence of steps. Moreover, studies have shown variability in surgeons' performances, which is valuable to expose residents to different ways to perform a procedure. However, it is unclear how to include the sequence of steps in training programs.

METHODS

We conducted a systematic review, including studies reporting explicit teaching of a standard sequence of steps, where assessment considered adherence to a standard sequence, and where faculty or students at any level participated. We searched for articles on PubMed, EMBASE, CINAHL, Web of Science, and Google Scholar databases.

RESULTS

We selected nine articles that met the inclusion criteria. The main strategy to teach the sequence was to use videos to demonstrate the procedure. The simulation was the main strategy to assess the learning of the sequence of steps. Non-standardized scoring protocols and written tests with variable validity evidence were the instruments used to assess the learning, and were focused on adherence to a standard sequence and the omission of steps.

CONCLUSIONS

Teaching and learning assessment of a standard sequence of steps is scarcely reported in procedural skills training literature. More research is needed to evaluate whether the new strategies to teach and assess the order of steps work. We recommend the use of Surgical Process Models and Surgical Data Science to incorporate the sequence of steps when teaching and assessing procedural skills.

Process-oriented metrics to provide feedback and assess the performance of students who are learning surgical procedures: The percutaneous dilatational tracheostomy case

PURPOSE

Assessing competency in surgical procedures is key for instructors to distinguish whether a resident is qualified to perform them on patients. Currently, assessment techniques do not always focus on providing feedback about the order in which the activities need to be performed. In this research, using a Process Mining approach, process-oriented metrics are proposed to assess the training of residents in a Percutaneous Dilatational Tracheostomy (PDT) simulator, identifying the critical points in the execution of the surgical process.

MATERIALS AND METHODS

A reference process model of the procedure was defined, and video recordings of student training sessions in the PDT simulator were collected and tagged to generate event logs. Three process-oriented metrics were proposed to assess the performance of the residents in training.

RESULTS

Although the students were proficient in classic metrics, they did not reach the optimum in process-oriented metrics. Only in 25% of the stages the optimum was achieved in the last session. In these stages, the four more challenging activities were also identified, which account for 32% of the process-oriented metrics errors.

CONCLUSIONS

Process-oriented metrics offer a new perspective on surgical procedures performance, providing a more granular perspective, which enables a more specific and actionable feedback for both students and instructors.

Surgical Process Modeling for Open Spinal Surgeries

Modern operating rooms are becoming increasingly advanced thanks to the emerging medical technologies and cutting-edge surgical techniques. Current surgeries are transitioning into complex processes that involve information and actions from multiple resources. When designing context-aware medical technologies for a given intervention, it is of utmost importance to have a deep understanding of the underlying surgical process. This is essential to develop technologies that can correctly address the clinical needs and can adapt to the existing workflow. Surgical Process Modeling (SPM) is a relatively recent discipline that focuses on achieving a profound understanding of the surgical workflow and providing a model that explains the elements of a given surgery as well as their sequence and hierarchy, both in quantitative and qualitative manner. To date, a significant body of work has been dedicated to the development of comprehensive SPMs for minimally invasive baroscopic and endoscopic surgeries, while such models are missing for open spinal surgeries. In this paper, we provide SPMs common open spinal interventions in orthopedics. Direct video observations of surgeries conducted in our institution were used to derive temporal and transitional information about the surgical activities. This information was later used to develop detailed SPMs that modeled different primary surgical steps and highlighted the frequency of transitions between the surgical activities made within each step. Given the recent emersion of advanced techniques that are tailored to open spinal surgeries (e.g., artificial intelligence methods for intraoperative guidance and navigation), we believe that the SPMs provided in this study can serve as the basis for further advancement of next-generation algorithms dedicated to open spinal interventions that require a profound understanding of the surgical workflow (e.g., automatic surgical activity recognition and surgical skill evaluation). Furthermore, the models provided in this study can potentially benefit the clinical community through standardization of the surgery, which is essential for surgical training.

Movement-level process modeling of microsurgical bimanual and unimanual tasks

Purpose

Microsurgical techniques require highly skilled manual handling of specialized surgical instruments. Surgical process models are central for objective evaluation of these skills, enabling data-driven solutions that can improve intraoperative efficiency.

Method

We built a surgical process model, defined at movement level in terms of elementary surgical actions (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=4$$\end{document}n=4) and targets (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=4$$\end{document}n=4). The model also included nonproductive movements, which enabled us to evaluate suturing efficiency and bi-manual dexterity. The elementary activities were used to investigate differences between novice (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document}n=5) and expert surgeons (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document}n=5) by comparing the cosine similarity of vector representations of a microsurgical suturing training task and its different segments.

Results

Based on our model, the experts were significantly more efficient than the novices at using their tools individually and simultaneously. At suture level, the experts were significantly more efficient at using their left hand tool, but the differences were not significant for the right hand tool. At the level of individual suture segments, the experts had on average 21.0 % higher suturing efficiency and 48.2 % higher bi-manual efficiency, and the results varied between segments. Similarity of the manual actions showed that expert and novice surgeons could be distinguished by their movement patterns.

Conclusions

The surgical process model allowed us to identify differences between novices’ and experts’ movements and to evaluate their uni- and bi-manual tool use efficiency. Analyzing surgical tasks in this manner could be used to evaluate surgical skill and help surgical trainees detect problems in their performance computationally.

How well do software assistants for minimally invasive partial nephrectomy meet surgeon information needs? A cognitive task analysis and literature review study

INTRODUCTION

Intraoperative software assistance is gaining increasing importance in laparoscopic and robot-assisted surgery. Within the user-centred development process of such systems, the first question to be asked is: What information does the surgeon need and when does he or she need it? In this article, we present an approach to investigate these surgeon information needs for minimally invasive partial nephrectomy and compare these needs to the relevant surgical computer assistance literature.

MATERIALS AND METHODS

First, we conducted a literature-based hierarchical task analysis of the surgical procedure. This task analysis was taken as a basis for a qualitative in-depth interview study with nine experienced surgical urologists. The study employed a cognitive task analysis method to elicit surgeons' information needs during minimally invasive partial nephrectomy. Finally, a systematic literature search was conducted to review proposed software assistance solutions for minimally invasive partial nephrectomy. The review focused on what information the solutions present to the surgeon and what phase of the surgery they aim to support.

RESULTS

The task analysis yielded a workflow description for minimally invasive partial nephrectomy. During the subsequent interview study, we identified three challenging phases of the procedure, which may particularly benefit from software assistance. These phases are I. Hilar and vascular management, II. Tumour excision, and III. Repair of the renal defects. Between these phases, 25 individual challenges were found which define the surgeon information needs. The literature review identified 34 relevant publications, all of which aim to support the surgeon in hilar and vascular management (phase I) or tumour excision (phase II).

CONCLUSION

The work presented in this article identified unmet surgeon information needs in minimally invasive partial nephrectomy. Namely, our results suggest that future solutions should address the repair of renal defects (phase III) or put more focus on the renal collecting system as a critical anatomical structure.

Enabling artificial intelligence in high acuity medical environments

Acute patient treatment can heavily profit from AI-based assistive and decision support systems, in terms of improved patient outcome as well as increased efficiency. Yet, only very few applications have been reported because of the limited accessibility of device data due to the lack of adoption of open standards, and the complexity of regulatory/approval requirements for AI-based systems. The fragmentation of data, still being stored in isolated silos, results in limited accessibility for AI in healthcare and machine learning is complicated by the loss of semantics in data conversions. We outline a reference model that addresses the requirements of innovative AI-based research systems as well as the clinical reality. The integration of networked medical devices and Clinical Repositories based on open standards, such as IEEE 11073 SDC and HL7 FHIR, will foster novel assistance and decision support. The reference model will make point-of-care device data available for AI-based approaches. Semantic interoperability between Clinical and Research Repositories will allow correlating patient data, device data, and the patient outcome. Thus, complete workflows in high acuity environments can be analysed. Open semantic interoperability will enable the improvement of patient outcome and the increase of efficiency on a large scale and across clinical applications.

Surgical process modelling strategies: which method to choose for determining workflow?

The vital role of surgeries in healthcare requires a constant attention to improvement. Surgical process modelling is an innovative and rather recently introduced approach for tackling the issues in today's complex surgeries. This modelling field is very challenging and still under development, therefore, it is not always clear which modelling strategy would best fit the needs in which situations. The aim of this study was to provide a guide for matching the choice of modelling strategies for determining surgical workflows. In this work, the concepts associated with surgical process modelling are described, aiming to clarify them and to promote their use in future studies. The relationship of these concepts and the possible combinations of the suitable approaches for modelling strategies are elaborated and the criteria for opting for the proper modelling strategy are discussed.