Intervention time prediction from surgical low-level tasks

A comparison between a maximum care university hospital and an outpatient clinic - potential for optimization in arthroscopic workflows?

BACKGROUND

Due to the growing economic pressure, there is an increasing interest in the optimization of operational processes within surgical operating rooms (ORs). Surgical departments are frequently dealing with limited resources, complex processes with unexpected events as well as constantly changing conditions. In order to use available resources efficiently, existing workflows and processes have to be analyzed and optimized continuously. Structural and procedural changes without prior data-driven analyses may impair the performance of the OR team and the overall efficiency of the department. The aim of this study is to develop an adaptable software toolset for surgical workflow analysis and perioperative process optimization in arthroscopic surgery.

METHODS

In this study, the perioperative processes of arthroscopic interventions have been recorded and analyzed subsequently. A total of 53 arthroscopic operations were recorded at a maximum care university hospital (UH) and 66 arthroscopic operations were acquired at a special outpatient clinic (OC). The recording includes regular perioperative processes (i.a. patient positioning, skin incision, application of wound dressing) and disruptive influences on these processes (e.g. telephone calls, missing or defective instruments, etc.). For this purpose, a software tool was developed ('s.w.an Suite Arthroscopic toolset'). Based on the data obtained, the processes of the maximum care provider and the special outpatient clinic have been analyzed in terms of performance measures (e.g. Closure-To-Incision-Time), efficiency (e.g. activity duration, OR resource utilization) as well as intra-process disturbances and then compared to one another.

RESULTS

Despite many similar processes, the results revealed considerable differences in performance indices. The OC required significantly less time than UH for surgical preoperative (UH: 30:47 min, OC: 26:01 min) and postoperative phase (UH: 15:04 min, OC: 9:56 min) as well as changeover time (UH: 32:33 min, OC: 6:02 min). In addition, these phases result in the Closure-to-Incision-Time, which lasted longer at the UH (UH: 80:01 min, OC: 41:12 min).

CONCLUSION

The perioperative process organization, team collaboration, and the avoidance of disruptive factors had a considerable influence on the progress of the surgeries. Furthermore, differences in terms of staffing and spatial capacities could be identified. Based on the acquired process data (such as the duration for different surgical steps or the number of interfering events) and the comparison of different arthroscopic departments, approaches for perioperative process optimization to decrease the time of work steps and reduce disruptive influences were identified.

Process model analysis of parenchyma sparing laparoscopic liver surgery to recognize surgical steps and predict impact of new technologies

BACKGROUND

Surgical process model (SPM) analysis is a great means to predict the surgical steps in a procedure as well as to predict the potential impact of new technologies. Especially in complicated and high-volume treatments, such as parenchyma sparing laparoscopic liver resection (LLR), profound process knowledge is essential for enabling improving surgical quality and efficiency.

METHODS

Videos of thirteen parenchyma sparing LLR were analyzed to extract the duration and sequence of surgical steps according to the process model. The videos were categorized into three groups, based on the tumor locations. Next, a detailed discrete events simulation model (DESM) of LLR was built, based on the process model and the process data obtained from the endoscopic videos. Furthermore, the impact of using a navigation platform on the total duration of the LLR was studied with the simulation model by assessing three different scenarios: (i) no navigation platform, (ii) conservative positive effect, and (iii) optimistic positive effect.

RESULTS

The possible variations of sequences of surgical steps in performing parenchyma sparing depending on the tumor locations were established. The statistically most probable chain of surgical steps was predicted, which could be used to improve parenchyma sparing surgeries. In all three categories (i-iii) the treatment phase covered the major part (~ 40%) of the total procedure duration (bottleneck). The simulation results predict that a navigation platform could decrease the total surgery duration by up to 30%.

CONCLUSION

This study showed a DESM based on the analysis of steps during surgical procedures can be used to predict the impact of new technology. SPMs can be used to detect, e.g., the most probable workflow paths which enables predicting next surgical steps, improving surgical training systems, and analyzing surgical performance. Moreover, it provides insight into the points for improvement and bottlenecks in the surgical process.

A Machine Learning Method for Automated Description and Workflow Analysis of First Trimester Ultrasound Scans

Obstetric ultrasound assessment of fetal anatomy in the first trimester of pregnancy is one of the less explored fields in obstetric sonography because of the paucity of guidelines on anatomical screening and availability of data. This paper, for the first time, examines imaging proficiency and practices of first trimester ultrasound scanning through analysis of full-length ultrasound video scans. Findings from this study provide insights to inform the development of more effective user-machine interfaces, of targeted assistive technologies, as well as improvements in workflow protocols for first trimester scanning. Specifically, this paper presents an automated framework to model operator clinical workflow from full-length routine first-trimester fetal ultrasound scan videos. The 2D+t convolutional neural network-based architecture proposed for video annotation incorporates transfer learning and spatio-temporal (2D+t) modelling to automatically partition an ultrasound video into semantically meaningful temporal segments based on the fetal anatomy detected in the video. The model results in a cross-validation A1 accuracy of 96.10% , F1=0.95 , precision =0.94 and recall =0.95 . Automated semantic partitioning of unlabelled video scans (n=250) achieves a high correlation with expert annotations ( ρ = 0.95, p=0.06 ). Clinical workflow patterns, operator skill and its variability can be derived from the resulting representation using the detected anatomy labels, order, and distribution. It is shown that nuchal translucency (NT) is the toughest standard plane to acquire and most operators struggle to localize high-quality frames. Furthermore, it is found that newly qualified operators spend 25.56% more time on key biometry tasks than experienced operators.

Analysing multi-perspective patient-related data during laparoscopic gynaecology procedures

Fusing data from different medical perspectives inside the operating room (OR) sets the stage for developing intelligent context-aware systems. These systems aim to promote better awareness inside the OR by keeping every medical team well informed about the work of other teams and thus mitigate conflicts resulting from different targets. In this research, a descriptive analysis of data collected from anaesthesiology and surgery was performed to investigate the relationships between the intra-abdominal pressure (IAP) and lung mechanics for patients during laparoscopic procedures. Data of nineteen patients who underwent laparoscopic gynaecology were included. Statistical analysis of all subjects showed a strong relationship between the IAP and dynamic lung compliance (r = 0.91). Additionally, the peak airway pressure was also strongly correlated to the IAP in volume-controlled ventilated patients (r = 0.928). Statistical results obtained by this study demonstrate the importance of analysing the relationship between surgical actions and physiological responses. Moreover, these results form the basis for developing medical decision support models, e.g., automatic compensation of IAP effects on lung function.

Generic surgical process model for minimally invasive liver treatment methods

Surgical process modelling is an innovative approach that aims to simplify the challenges involved in improving surgeries through quantitative analysis of a well-established model of surgical activities. In this paper, surgical process model strategies are applied for the analysis of different Minimally Invasive Liver Treatments (MILTs), including ablation and surgical resection of the liver lesions. Moreover, a generic surgical process model for these differences in MILTs is introduced. The generic surgical process model was established at three different granularity levels. The generic process model, encompassing thirteen phases, was verified against videos of MILT procedures and interviews with surgeons. The established model covers all the surgical and interventional activities and the connections between them and provides a foundation for extensive quantitative analysis and simulations of MILT procedures for improving computer-assisted surgery systems, surgeon training and evaluation, surgeon guidance and planning systems and evaluation of new technologies.

Convolutional-de-convolutional neural networks for recognition of surgical workflow

Computer-assisted surgery (CAS) has occupied an important position in modern surgery, further stimulating the progress of methodology and technology. In recent years, a large number of computer vision-based methods have been widely used in surgical workflow recognition tasks. For training the models, a lot of annotated data are necessary. However, the annotation of surgical data requires expert knowledge and thus becomes difficult and time-consuming. In this paper, we focus on the problem of data deficiency and propose a knowledge transfer learning method based on artificial neural network to compensate a small amount of labeled training data. To solve this problem, we propose an unsupervised method for pre-training a Convolutional-De-Convolutional (CDC) neural network for sequencing surgical workflow frames, which performs neural convolution in space (for semantic abstraction) and neural de-convolution in time (for frame level resolution) simultaneously. Specifically, through neural convolution transfer learning, we only fine-tuned the CDC neural network to classify the surgical phase. We performed some experiments for validating the model, and it showed that the proposed model can effectively extract the surgical feature and determine the surgical phase. The accuracy (Acc), recall, precision (Pres) of our model reached 91.4, 78.9, and 82.5%, respectively.

Clinical study of skill assessment based on time sequential measurement changes

Endoscopic sinus surgery is a common procedure for chronic sinusitis; however, complications have been reported in some cases. Improving surgical outcomes requires an improvement in a surgeon's skills. In this study, we used surgical workflow analysis to automatically extract "errors," indicating whether there was a large difference in the comparative evaluation of procedures performed by experts and residents. First, we quantified surgical features using surgical log data, which contained surgical instrument information (e.g., tip position) and time stamp. Second, we created a surgical process model (SPM), which represents the temporal transition of the surgical features. Finally, we identified technical issues by creating an expert standard SPM and comparing it to the novice SPM. We verified the performance of our methods by using the clinical data of 39 patients. In total, 303 portions were detected as an error, and they were classified into six categories. Three risky operations were overlooked, and there were 11 overdetected errors. We noted that most errors detected by our method involved dangers. The implementation of our methods of automatic improvement points detection may be advantageous. Our methods may help reduce the time for reviewing and improving the surgical technique efficiently.

State-of-the-art of situation recognition systems for intraoperative procedures

One of the key challenges for automatic assistance is the support of actors in the operating room depending on the status of the procedure. Therefore, context information collected in the operating room is used to gain knowledge about the current situation. In literature, solutions already exist for specific use cases, but it is doubtful to what extent these approaches can be transferred to other conditions. We conducted a comprehensive literature research on existing situation recognition systems for the intraoperative area, covering 274 articles and 95 cross-references published between 2010 and 2019. We contrasted and compared 58 identified approaches based on defined aspects such as used sensor data or application area. In addition, we discussed applicability and transferability. Most of the papers focus on video data for recognizing situations within laparoscopic and cataract surgeries. Not all of the approaches can be used online for real-time recognition. Using different methods, good results with recognition accuracies above 90% could be achieved. Overall, transferability is less addressed. The applicability of approaches to other circumstances seems to be possible to a limited extent. Future research should place a stronger focus on adaptability. The literature review shows differences within existing approaches for situation recognition and outlines research trends. Applicability and transferability to other conditions are less addressed in current work.

Surgical Process Modeling for Open Spinal Surgeries

Modern operating rooms are becoming increasingly advanced thanks to the emerging medical technologies and cutting-edge surgical techniques. Current surgeries are transitioning into complex processes that involve information and actions from multiple resources. When designing context-aware medical technologies for a given intervention, it is of utmost importance to have a deep understanding of the underlying surgical process. This is essential to develop technologies that can correctly address the clinical needs and can adapt to the existing workflow. Surgical Process Modeling (SPM) is a relatively recent discipline that focuses on achieving a profound understanding of the surgical workflow and providing a model that explains the elements of a given surgery as well as their sequence and hierarchy, both in quantitative and qualitative manner. To date, a significant body of work has been dedicated to the development of comprehensive SPMs for minimally invasive baroscopic and endoscopic surgeries, while such models are missing for open spinal surgeries. In this paper, we provide SPMs common open spinal interventions in orthopedics. Direct video observations of surgeries conducted in our institution were used to derive temporal and transitional information about the surgical activities. This information was later used to develop detailed SPMs that modeled different primary surgical steps and highlighted the frequency of transitions between the surgical activities made within each step. Given the recent emersion of advanced techniques that are tailored to open spinal surgeries (e.g., artificial intelligence methods for intraoperative guidance and navigation), we believe that the SPMs provided in this study can serve as the basis for further advancement of next-generation algorithms dedicated to open spinal interventions that require a profound understanding of the surgical workflow (e.g., automatic surgical activity recognition and surgical skill evaluation). Furthermore, the models provided in this study can potentially benefit the clinical community through standardization of the surgery, which is essential for surgical training.

Movement-level process modeling of microsurgical bimanual and unimanual tasks

Purpose

Microsurgical techniques require highly skilled manual handling of specialized surgical instruments. Surgical process models are central for objective evaluation of these skills, enabling data-driven solutions that can improve intraoperative efficiency.

Method

We built a surgical process model, defined at movement level in terms of elementary surgical actions (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=4$$\end{document}n=4) and targets (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=4$$\end{document}n=4). The model also included nonproductive movements, which enabled us to evaluate suturing efficiency and bi-manual dexterity. The elementary activities were used to investigate differences between novice (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document}n=5) and expert surgeons (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document}n=5) by comparing the cosine similarity of vector representations of a microsurgical suturing training task and its different segments.

Results

Based on our model, the experts were significantly more efficient than the novices at using their tools individually and simultaneously. At suture level, the experts were significantly more efficient at using their left hand tool, but the differences were not significant for the right hand tool. At the level of individual suture segments, the experts had on average 21.0 % higher suturing efficiency and 48.2 % higher bi-manual efficiency, and the results varied between segments. Similarity of the manual actions showed that expert and novice surgeons could be distinguished by their movement patterns.

Conclusions

The surgical process model allowed us to identify differences between novices’ and experts’ movements and to evaluate their uni- and bi-manual tool use efficiency. Analyzing surgical tasks in this manner could be used to evaluate surgical skill and help surgical trainees detect problems in their performance computationally.

Knowledge representation and learning of operator clinical workflow from full-length routine fetal ultrasound scan videos

Ultrasound is a widely used imaging modality, yet it is well-known that scanning can be highly operator-dependent and difficult to perform, which limits its wider use in clinical practice. The literature on understanding what makes clinical sonography hard to learn and how sonography varies in the field is sparse, restricted to small-scale studies on the effectiveness of ultrasound training schemes, the role of ultrasound simulation in training, and the effect of introducing scanning guidelines and standards on diagnostic image quality. The Big Data era, and the recent and rapid emergence of machine learning as a more mainstream large-scale data analysis technique, presents a fresh opportunity to study sonography in the field at scale for the first time. Large-scale analysis of video recordings of full-length routine fetal ultrasound scans offers the potential to characterise differences between the scanning proficiency of experts and trainees that would be tedious and time-consuming to do manually due to the vast amounts of data. Such research would be informative to better understand operator clinical workflow when conducting ultrasound scans to support skills training, optimise scan times, and inform building better user-machine interfaces. This paper is to our knowledge the first to address sonography data science, which we consider in the context of second-trimester fetal sonography screening. Specifically, we present a fully-automatic framework to analyse operator clinical workflow solely from full-length routine second-trimester fetal ultrasound scan videos. An ultrasound video dataset containing more than 200 hours of scan recordings was generated for this study. We developed an original deep learning method to temporally segment the ultrasound video into semantically meaningful segments (the video description). The resulting semantic annotation was then used to depict operator clinical workflow (the knowledge representation). Machine learning was applied to the knowledge representation to characterise operator skills and assess operator variability. For video description, our best-performing deep spatio-temporal network shows favourable results in cross-validation (accuracy: 91.7%), statistical analysis (correlation: 0.98, p < 0.05) and retrospective manual validation (accuracy: 76.4%). For knowledge representation of operator clinical workflow, a three-level abstraction scheme consisting of a Subject-specific Timeline Model (STM), Summary of Timeline Features (STF), and an Operator Graph Model (OGM), was introduced that led to a significant decrease in dimensionality and computational complexity compared to raw video data. The workflow representations were learnt to discriminate between operator skills, where a proposed convolutional neural network-based model showed most promising performance (cross-validation accuracy: 98.5%, accuracy on unseen operators: 76.9%). These were further used to derive operator-specific scanning signatures and operator variability in terms of type, order and time distribution of constituent tasks.

Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery

Purpose

In the context of aviation and automotive navigation technology, assistance functions are associated with predictive planning and wayfinding tasks. In endoscopic minimally invasive surgery, however, assistance so far relies primarily on image-based localization and classification. We show that navigation workflows can be described and used for the prediction of navigation steps.

Methods

A natural description vocabulary for observable anatomical landmarks in endoscopic images was defined to create 3850 navigation workflow sentences from 22 annotated functional endoscopic sinus surgery (FESS) recordings. Resulting FESS navigation workflows showed an imbalanced data distribution with over-represented landmarks in the ethmoidal sinus. A transformer model was trained to predict navigation sentences in sequence-to-sequence tasks. The training was performed with the Adam optimizer and label smoothing in a leave-one-out cross-validation study. The sentences were generated using an adapted beam search algorithm with exponential decay beam rescoring. The transformer model was compared to a standard encoder-decoder-model, as well as HMM and LSTM baseline models.

Results

The transformer model reached the highest prediction accuracy for navigation steps at 0.53, followed by 0.35 of the LSTM and 0.32 for the standard encoder-decoder-network. With an accuracy of sentence generation of 0.83, the prediction of navigation steps at sentence-level benefits from the additional semantic information. While standard class representation predictions suffer from an imbalanced data distribution, the attention mechanism also considered underrepresented classes reasonably well.

Conclusion

We implemented a natural language-based prediction method for sentence-level navigation steps in endoscopic surgery. The sentence-level prediction method showed a potential that word relations to navigation tasks can be learned and used for predicting future steps. Further studies are needed to investigate the functionality of path prediction. The prediction approach is a first step in the field of visuo-linguistic navigation assistance for endoscopic minimally invasive surgery.

Resilience in the Surgical Scheduling to Support Adaptive Scheduling System

Operating Room (OR) managers frequently encounter uncertainties related to real-time scheduling, especially on the day of surgery. It is necessary to enable earlier identification of uncertainties occurring in the perioperative environment. This study aims to propose a framework for resilient surgical scheduling by identifying uncertainty factors affecting the real-time surgical scheduling through a mixed-methods study. We collected the pre- and post-surgical scheduling data for twenty days and a one-day observation data in a top-tier general university hospital in South Korea. Data were compared and analyzed for any changes related to the dimensions of uncertainty. The observations in situ of surgical scheduling were performed to confirm our findings from the quantitative data. Analysis was divided into two phases of fundamental uncertainties categorization (conceptual, technical and personal) and uncertainties leveling for effective decision-making strategies. Pre- and post-surgical scheduling data analysis showed that unconfirmed patient medical conditions and emergency cases are the main causes of frequent same-day surgery schedule changes, with derived factors that affect the scheduling pattern (time of surgery, overtime surgery, surgical procedure changes and surgery duration). The observation revealed how the OR manager controlled the unexpected events to prevent overtime surgeries. In conclusion, integrating resilience approach to identifying uncertainties and managing event changes can minimize potential risks that may compromise the surgical personnel and patients' safety, thereby promoting higher resilience in the current system. Furthermore, this strategy may improve coordination among personnel and increase surgical scheduling efficiency.

RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations

Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. The previous state-of-the-art approaches for RSD prediction are dependent on manual annotation, whose generation requires expensive expert knowledge and is time-consuming, especially considering the numerous types of surgeries performed in a hospital and the large number of laparoscopic videos available. A crucial feature of RSDNet is that it does not depend on any manual annotation during training, making it easily scalable to many kinds of surgeries. The generalizability of our approach is demonstrated by testing the pipeline on two large datasets containing different types of surgeries: 120 cholecystectomy and 170 gastric bypass videos. The experimental results also show that the proposed network significantly outperforms a traditional method of estimating RSD without utilizing manual annotation. Further, this paper provides a deeper insight into the deep learning network through visualization and interpretation of the features that are automatically learned.

Surgical process modelling strategies: which method to choose for determining workflow?

The vital role of surgeries in healthcare requires a constant attention to improvement. Surgical process modelling is an innovative and rather recently introduced approach for tackling the issues in today's complex surgeries. This modelling field is very challenging and still under development, therefore, it is not always clear which modelling strategy would best fit the needs in which situations. The aim of this study was to provide a guide for matching the choice of modelling strategies for determining surgical workflows. In this work, the concepts associated with surgical process modelling are described, aiming to clarify them and to promote their use in future studies. The relationship of these concepts and the possible combinations of the suitable approaches for modelling strategies are elaborated and the criteria for opting for the proper modelling strategy are discussed.

Surgical process modeling

Due to the rapidly evolving medical, technological, and technical possibilities, surgical procedures are becoming more and more complex. On the one hand, this offers an increasing number of advantages for patients, such as enhanced patient safety, minimal invasive interventions, and less medical malpractices. On the other hand, it also heightens pressure on surgeons and other clinical staff and has brought about a new policy in hospitals, which must rely on a great number of economic, social, psychological, qualitative, practical, and technological resources. As a result, medical disciplines, such as surgery, are slowly merging with technical disciplines. However, this synergy is not yet fully matured. The current information and communication technology in hospitals cannot manage the clinical and operational sequence adequately. The consequences are breaches in the surgical workflow, extensions in procedure times, and media disruptions. Furthermore, the data accrued in operating rooms (ORs) by surgeons and systems are not sufficiently implemented. A flood of information, “big data”, is available from information systems. That might be deployed in the context of Medicine 4.0 to facilitate the surgical treatment. However, it is unused due to infrastructure breaches or communication errors. Surgical process models (SPMs) alleviate these problems. They can be defined as simplified, formal, or semiformal representations of a network of surgery-related activities, reflecting a predefined subset of interest. They can employ different means of generation, languages, and data acquisition strategies. They can represent surgical interventions with high resolution, offering qualifiable and quantifiable information on the course of the intervention on the level of single, minute, surgical work-steps. The basic idea is to gather information concerning the surgical intervention and its activities, such as performance time, surgical instrument used, trajectories, movements, or intervention phases. These data can be gathered by means of workflow recordings. These recordings are abstracted to represent an individual surgical process as a model and are an essential requirement to enable Medicine 4.0 in the OR. Further abstraction can be generated by merging individual process models to form generic SPMs to increase the validity for a larger number of patients. Furthermore, these models can be applied in a wide variety of use-cases. In this regard, the term “modeling” can be used to support either one or more of the following tasks: “to describe”, “to understand”, “to explain”, to optimize”, “to learn”, “to teach”, or “to automate”. Possible use-cases are requirements analyses, evaluating surgical assist systems, generating surgeon-specific training-recommendation, creating workflow management systems for ORs, and comparing different surgical strategies. The presented chapter will give an introduction into this challenging topic, presenting different methods to generate SPMs from the workflow in the OR, as well as various use-cases, and state-of-the-art research in this field. Although many examples in the article are given according to SPMs that were computed based on observations, the same approaches can be easily applied to SPMs that were measured automatically and mined from big data.

Automatic matching of surgeries to predict surgeons' next actions

OBJECTIVE

More than half a million surgeries are performed every day worldwide, which makes surgery one of the most important component of global health care. In this context, the objective of this paper is to introduce a new method for the prediction of the possible next task that a surgeon is going to perform during surgery.

MATERIAL AND METHOD

We formulate the problem as finding the optimal registration of a partial sequence to a complete reference sequence of surgical activities. We propose an efficient algorithm to find the optimal partial alignment and a prediction system using maximum a posteriori probability estimation and filtering. We also introduce a weighting scheme allowing to improve the predictions by taking into account the relative similarity between the current surgery and a set of pre-recorded surgeries.

RESULTS

Our method is evaluated on two types of neurosurgical procedures: lumbar disc herniation removal and anterior cervical discectomy. Results show that our method outperformed the state of the art by predicting the next task that the surgeon will perform with 95% accuracy.

CONCLUSIONS

This work shows that, even from the low-level description of surgeries and without other sources of information, it is often possible to predict the next surgical task when the conditions are consistent with the previously recorded surgeries. We also showed that our method is able to assess when there is actually a large divergence between the predictions and decide that it is not reasonable to make a prediction.

Rule-based medical device adaptation for the digital operating room

A workflow-driven cooperative operating room needs to be established in order to successfully unburden the surgeon and the operating room staff very time-consuming information-seeking and configuration tasks. We propose an approach towards the integration of intraoperative surgical workflow management and integration technologies. The concept of rule-based behavior is adapted to situation-aware medical devices. A prototype was implemented and experiments with sixty recorded brain tumor removal procedures were conducted to test the proposed approach. An analysis of the recordings indicated numerous applications, such as automatic display configuration, room light adaptation and pre-configuration of medical devices and systems.

Online time and resource management based on surgical workflow time series analysis

PURPOSE

Hospitals' effectiveness and efficiency can be enhanced by automating the resource and time management of the most cost-intensive unit in the hospital: the operating room (OR). The key elements required for the ideal organization of hospital staff and technical resources (such as instruments in the OR) are an exact online forecast of both the surgeon's resource usage and the remaining intervention time.

METHODS

This paper presents a novel online approach relying on time series analysis and the application of a linear time-variant system. We calculated the power spectral density and the spectrogram of surgical perspectives (e.g., used instrument) of interest to compare several surgical workflows.

RESULTS

Considering only the use of the surgeon's right hand during an intervention, we were able to predict the remaining intervention time online with an error of 21 min 45 s ±9 min 59 s for lumbar discectomy. Furthermore, the performance of forecasting of technical resource usage in the next 20 min was calculated for a combination of spectral analysis and the application of a linear time-variant system (sensitivity: 74 %; specificity: 75 %) focusing on just the use of surgeon's instrument in question.

CONCLUSION

The outstanding benefit of these methods is that the automated recording of surgical workflows has minimal impact during interventions since the whole set of surgical perspectives need not be recorded. The resulting predictions can help various stakeholders such as OR staff and hospital technicians. Moreover, reducing resource conflicts could well improve patient care.

Towards structuring contextual information for workflow-driven surgical assistance functionalities

A workflow-driven cooperative working environment needs to be established in order to successfully unburden the surgeon and the OR staff from technical configuration and information-seeking tasks. An important prerequisite for autonomous situationaware adaptation of medical devices is a comprehensive representation of the operating context regarding the surgical process and situation.

We propose a hierarchical structuring of process-related and situation-related information entities and include assessment scores that intraoperative workflow information systems may provide via OR networks. The conducted experiments on the proposed assessment scores included sixty recorded brain tumour removal procedures and considered 344 distinguishable surgical situations.

A comprehensive modelling of surgical situations and process context will be a significant pre-requisite for reliable autonomous adaptation of medical devices and systems in digital operating rooms.

Frequency based assessment of surgical activities

In hospitals the duration of surgeries plays a decisive role in many areas, such as patient safety or financial aspects. By utilizing accurate automated online prediction efficient surgical patient care and effective resource management can be attained. In this work several surgical activities during an intervention were examined for their potential to forecast the remaining intervention time. The method used was based on analysing in the frequency domain of time series which represented the status of surgical activities during an intervention. A nonparametric estimation of power spectral density was calculated for single surgical tasks during an intervention. The power spectral densities (PSD) of different surgical activities were compared in a leave-one-out cross validation of forty surgical workflow recordings of lumbar discectomies. The results showed that the activity irrigate with a mean prediction error of 26 min 23 s is best-suited for determining the remainder of the intervention. To construct a scheduling support for a wider range of surgery types the actions conducted by the surgeon’s right and left hand would eminently be more suitable; the error of the action right hand was 41 min 39 s, yet. In conclusion sophistication into the presented frequency based method might support time and resource management in a general manner.

Multi-perspective workflow modeling for online surgical situation models

INTRODUCTION

Surgical workflow management is expected to enable situation-aware adaptation and intelligent systems behavior in an integrated operating room (OR). The overall aim is to unburden the surgeon and OR staff from both manual maintenance and information seeking tasks. A major step toward intelligent systems behavior is a stable classification of the surgical situation from multiple perspectives based on performed low-level tasks.

MATERIAL AND METHODS

The present work proposes a method for the classification of surgical situations based on multi-perspective workflow modeling. A model network that interconnects different types of surgical process models is described. Various aspects of a surgical situation description were considered: low-level tasks, high-level tasks, patient status, and the use of medical devices. A study with sixty neurosurgical interventions was conducted to evaluate the performance of our approach and its robustness against incomplete workflow recognition input.

RESULTS

A correct classification rate of over 90% was measured for high-level tasks and patient status. The device usage models for navigation and neurophysiology classified over 95% of the situations correctly, whereas the ultrasound usage was more difficult to predict. Overall, the classification rate decreased with an increasing level of input distortion.

DISCUSSION

Autonomous adaptation of medical devices and intelligent systems behavior do not currently depend solely on low-level tasks. Instead, they require a more general type of understanding of the surgical condition. The integration of various surgical process models in a network provided a comprehensive representation of the interventions and allowed for the generation of extensive situation descriptions.

CONCLUSION

Multi-perspective surgical workflow modeling and online situation models will be a significant pre-requisite for reliable and intelligent systems behavior. Hence, they will contribute to a cooperative OR environment.

Vision-based online recognition of surgical activities

PURPOSE

Surgical processes are complex entities characterized by expressive models and data. Recognizable activities define each surgical process. The principal limitation of current vision-based recognition methods is inefficiency due to the large amount of information captured during a surgical procedure. To overcome this technical challenge, we introduce a surgical gesture recognition system using temperature-based recognition.

METHODS

An infrared thermal camera was combined with a hierarchical temporal memory and was used during surgical procedures. The recordings were analyzed for recognition of surgical activities. The image sequence information acquired included hand temperatures. This datum was analyzed to perform gesture extraction and recognition based on heat differences between the surgeon's warm hands and the colder background of the environment.

RESULTS

The system was validated by simulating a functional endoscopic sinus surgery, a common type of otolaryngologic surgery. The thermal camera was directed toward the hands of the surgeon while handling different instruments. The system achieved an online recognition accuracy of 96% with high precision and recall rates of approximately 60%.

CONCLUSION

Vision-based recognition methods are the current best practice approaches for monitoring surgical processes. Problems of information overflow and extended recognition times in vision-based approaches were overcome by changing the spectral range to infrared. This change enables the real-time recognition of surgical activities and provides online monitoring information to surgical assistance systems and workflow management systems.

Requirements for the structured recording of surgical device data in the digital operating room

PURPOSE

Due to the increasing complexity of the surgical working environment, increasingly technical solutions must be found to help relieve the surgeon. This objective is supported by a structured storage concept for all relevant device data.

METHODS

In this work, we present a concept and prototype development of a storage system to address intraoperative medical data. The requirements of such a system are described, and solutions for data transfer, processing, and storage are presented. In a subsequent study, a prototype based on the presented concept is tested for correct and complete data transmission and storage and for the ability to record a complete neurosurgical intervention with low processing latencies. In the final section, several applications for the presented data recorder are shown.

RESULTS

The developed system based on the presented concept is able to store the generated data correctly, completely, and quickly enough even if much more data than expected are sent during a surgical intervention.

CONCLUSIONS

The Surgical Data Recorder supports automatic recognition of the interventional situation by providing a centralized data storage and access interface to the OR communication bus. In the future, further data acquisition technologies should be integrated. Therefore, additional interfaces must be developed. The data generated by these devices and technologies should also be stored in or referenced by the Surgical Data Recorder to support the analysis of the OR situation.