Toward a standard ontology of surgical process models

An ontology-based tool for modeling and documenting events in neurosurgery

BACKGROUND

Intraoperative neurophysiological monitoring (IOM) plays a pivotal role in enhancing patient safety during neurosurgical procedures. This vital technique involves the continuous measurement of evoked potentials to provide early warnings and ensure the preservation of critical neural structures. One of the primary challenges has been the effective documentation of IOM events with semantically enriched characterizations. This study aimed to address this challenge by developing an ontology-based tool.

METHODS

We structured the development of the IOM Documentation Ontology (IOMDO) and the associated tool into three distinct phases. The initial phase focused on the ontology's creation, drawing from the OBO (Open Biological and Biomedical Ontology) principles. The subsequent phase involved agile software development, a flexible approach to encapsulate the diverse requirements and swiftly produce a prototype. The last phase entailed practical evaluation within real-world documentation settings. This crucial stage enabled us to gather firsthand insights, assessing the tool's functionality and efficacy. The observations made during this phase formed the basis for essential adjustments to ensure the tool's productive utilization.

RESULTS

The core entities of the ontology revolve around central aspects of IOM, including measurements characterized by timestamp, type, values, and location. Concepts and terms of several ontologies were integrated into IOMDO, e.g., the Foundation Model of Anatomy (FMA), the Human Phenotype Ontology (HPO) and the ontology for surgical process models (OntoSPM) related to general surgical terms. The software tool developed for extending the ontology and the associated knowledge base was built with JavaFX for the user-friendly frontend and Apache Jena for the robust backend. The tool's evaluation involved test users who unanimously found the interface accessible and usable, even for those without extensive technical expertise.

CONCLUSIONS

Through the establishment of a structured and standardized framework for characterizing IOM events, our ontology-based tool holds the potential to enhance the quality of documentation, benefiting patient care by improving the foundation for informed decision-making. Furthermore, researchers can leverage the semantically enriched data to identify trends, patterns, and areas for surgical practice enhancement. To optimize documentation through ontology-based approaches, it's crucial to address potential modeling issues that are associated with the Ontology of Adverse Events.

Autonomous System for Tumor Resection (ASTR) - Dual-Arm Robotic Midline Partial Glossectomy

Head and neck cancers are the seventh most common cancers worldwide, with squamous cell carcinoma being the most prevalent histologic subtype. Surgical resection is a primary treatment modality for many patients with head and neck squamous cell carcinoma, and accurately identifying tumor boundaries and ensuring sufficient resection margins are critical for optimizing oncologic outcomes. This study presents an innovative autonomous system for tumor resection (ASTR) and conducts a feasibility study by performing supervised autonomous midline partial glossectomy for pseudotumor with millimeter accuracy. The proposed ASTR system consists of a dual-camera vision system, an electrosurgical instrument, a newly developed vacuum grasping instrument, two 6-DOF manipulators, and a novel autonomous control system. The letter introduces an ontology-based research framework for creating and implementing a complex autonomous surgical workflow, using the glossectomy as a case study. Porcine tongue tissues are used in this study, and marked using color inks and near-infrared fluorescent (NIRF) markers to indicate the pseudotumor. ASTR actively monitors the NIRF markers and gathers spatial and color data from the samples, enabling planning and execution of robot trajectories in accordance with the proposed glossectomy workflow. The system successfully performs six consecutive supervised autonomous pseudotumor resections on porcine specimens. The average surface and depth resection errors measure 0.73±0.60 mm and 1.89±0.54 mm, respectively, with no positive tumor margins detected in any of the six resections. The resection accuracy is demonstrated to be on par with manual pseudotumor glossectomy performed by an experienced otolaryngologist.

COMPASS: a formal framework and aggregate dataset for generalized surgical procedure modeling

PURPOSE

We propose a formal framework for the modeling and segmentation of minimally invasive surgical tasks using a unified set of motion primitives (MPs) to enable more objective labeling and the aggregation of different datasets.

METHODS

We model dry-lab surgical tasks as finite state machines, representing how the execution of MPs as the basic surgical actions results in the change of surgical context, which characterizes the physical interactions among tools and objects in the surgical environment. We develop methods for labeling surgical context based on video data and for automatic translation of context to MP labels. We then use our framework to create the COntext and Motion Primitive Aggregate Surgical Set (COMPASS), including six dry-lab surgical tasks from three publicly available datasets (JIGSAWS, DESK, and ROSMA), with kinematic and video data and context and MP labels.

RESULTS

Our context labeling method achieves near-perfect agreement between consensus labels from crowd-sourcing and expert surgeons. Segmentation of tasks to MPs results in the creation of the COMPASS dataset that nearly triples the amount of data for modeling and analysis and enables the generation of separate transcripts for the left and right tools.

CONCLUSION

The proposed framework results in high quality labeling of surgical data based on context and fine-grained MPs. Modeling surgical tasks with MPs enables the aggregation of different datasets and the separate analysis of left and right hands for bimanual coordination assessment. Our formal framework and aggregate dataset can support the development of explainable and multi-granularity models for improved surgical process analysis, skill assessment, error detection, and autonomy.

Surgical Phase Duration in Robot-Assisted Partial Nephrectomy: A Surgical Data Science Exploration for Clinical Relevance

(1) Background: Surgical phases form the basic building blocks for surgical skill assessment, feedback, and teaching. The phase duration itself and its correlation with clinical parameters at diagnosis have not yet been investigated. Novel commercial platforms provide phase indications but have not been assessed for accuracy yet. (2) Methods: We assessed 100 robot-assisted partial nephrectomy videos for phase durations based on previously defined proficiency metrics. We developed an annotation framework and subsequently compared our annotations to an existing commercial solution (Touch Surgery, Medtronic™). We subsequently explored clinical correlations between phase durations and parameters derived from diagnosis and treatment. (3) Results: An objective and uniform phase assessment requires precise definitions derived from an iterative revision process. A comparison to a commercial solution shows large differences in definitions across phases. BMI and the duration of renal tumor identification are positively correlated, as are tumor complexity and both tumor excision and renorrhaphy duration. (4) Conclusions: The surgical phase duration can be correlated with certain clinical outcomes. Further research should investigate whether the retrieved correlations are also clinically meaningful. This requires an increase in dataset sizes and facilitation through intelligent computer vision algorithms. Commercial platforms can facilitate this dataset expansion and help unlock the full potential, provided that the phase annotation details are disclosed.

SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education)

BACKGROUND

Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various factors, including data structure, acquisition, storage, and sharing; data use and exploration, and finally data governance, which encompasses all ethical and legal regulations associated with the data. There is a universal need among stakeholders in surgical data science to establish standardized frameworks that address all aspects of this lifecycle to ensure data quality and purpose.

METHODS

Working groups were formed, among 48 representatives from academia and industry, including clinicians, computer scientists and industry representatives. These working groups focused on: Data Use, Data Structure, Data Exploration, and Data Governance. After working group and panel discussions, a modified Delphi process was conducted.

RESULTS

The resulting Delphi consensus provides conceptualized and structured recommendations for each domain related to surgical video data. We identified the key stakeholders within the data lifecycle and formulated comprehensive, easily understandable, and widely applicable guidelines for data utilization. Standardization of data structure should encompass format and quality, data sources, documentation, metadata, and account for biases within the data. To foster scientific data exploration, datasets should reflect diversity and remain adaptable to future applications. Data governance must be transparent to all stakeholders, addressing legal and ethical considerations surrounding the data.

CONCLUSION

This consensus presents essential recommendations around the generation of standardized and diverse surgical video databanks, accounting for multiple stakeholders involved in data generation and use throughout its lifecycle. Following the SAGES annotation framework, we lay the foundation for standardization of data use, structure, and exploration. A detailed exploration of requirements for adequate data governance will follow.

CholecTriplet2022: Show me a tool and tell me the triplet - An endoscopic vision challenge for surgical action triplet detection

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of ‹instrument, verb, target› triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

A methodology for the annotation of surgical videos for supervised machine learning applications

PURPOSE

Surgical data science is an emerging field focused on quantitative analysis of pre-, intra-, and postoperative patient data (Maier-Hein et al. in Med Image Anal 76: 102306, 2022). Data science approaches can decompose complex procedures, train surgical novices, assess outcomes of actions, and create predictive models of surgical outcomes (Marcus et al. in Pituitary 24: 839-853, 2021; Røadsch et al. in Nat Mach Intell, 2022). Surgical videos contain powerful signals of events that may impact patient outcomes. A necessary step before the deployment of supervised machine learning methods is the development of labels for objects and anatomy. We describe a complete method for annotating videos of transsphenoidal surgery.

METHODS

Endoscopic video recordings of transsphenoidal pituitary tumor removal surgeries were collected from a multicenter research collaborative. These videos were anonymized and stored in a cloud-based platform. Videos were uploaded to an online annotation platform. Annotation framework was developed based on a literature review and surgical observations to ensure proper understanding of the tools, anatomy, and steps present. A user guide was developed to trained annotators to ensure standardization.

RESULTS

A fully annotated video of a transsphenoidal pituitary tumor removal surgery was produced. This annotated video included over 129,826 frames. To prevent any missing annotations, all frames were later reviewed by highly experienced annotators and a surgeon reviewer. Iterations to annotated videos allowed for the creation of an annotated video complete with labeled surgical tools, anatomy, and phases. In addition, a user guide was developed for the training of novice annotators, which provides information about the annotation software to ensure the production of standardized annotations.

CONCLUSIONS

A standardized and reproducible workflow for managing surgical video data is a necessary prerequisite to surgical data science applications. We developed a standard methodology for annotating surgical videos that may facilitate the quantitative analysis of videos using machine learning applications. Future work will demonstrate the clinical relevance and impact of this workflow by developing process modeling and outcome predictors.

A systematic review of annotation for surgical process model analysis in minimally invasive surgery based on video

BACKGROUND

Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science. The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos.

METHODS

For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool.

RESULTS

Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies.

CONCLUSION

Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.

Artificial intelligence and automation in endoscopy and surgery

Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patient's anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery.

Proposal and multicentric validation of a laparoscopic Roux-en-Y gastric bypass surgery ontology

BACKGROUND

Phase and step annotation in surgical videos is a prerequisite for surgical scene understanding and for downstream tasks like intraoperative feedback or assistance. However, most ontologies are applied on small monocentric datasets and lack external validation. To overcome these limitations an ontology for phases and steps of laparoscopic Roux-en-Y gastric bypass (LRYGB) is proposed and validated on a multicentric dataset in terms of inter- and intra-rater reliability (inter-/intra-RR).

METHODS

The proposed LRYGB ontology consists of 12 phase and 46 step definitions that are hierarchically structured. Two board certified surgeons (raters) with > 10 years of clinical experience applied the proposed ontology on two datasets: (1) StraBypass40 consists of 40 LRYGB videos from Nouvel Hôpital Civil, Strasbourg, France and (2) BernBypass70 consists of 70 LRYGB videos from Inselspital, Bern University Hospital, Bern, Switzerland. To assess inter-RR the two raters' annotations of ten randomly chosen videos from StraBypass40 and BernBypass70 each, were compared. To assess intra-RR ten randomly chosen videos were annotated twice by the same rater and annotations were compared. Inter-RR was calculated using Cohen's kappa. Additionally, for inter- and intra-RR accuracy, precision, recall, F1-score, and application dependent metrics were applied.

RESULTS

The mean ± SD video duration was 108 ± 33 min and 75 ± 21 min in StraBypass40 and BernBypass70, respectively. The proposed ontology shows an inter-RR of 96.8 ± 2.7% for phases and 85.4 ± 6.0% for steps on StraBypass40 and 94.9 ± 5.8% for phases and 76.1 ± 13.9% for steps on BernBypass70. The overall Cohen's kappa of inter-RR was 95.9 ± 4.3% for phases and 80.8 ± 10.0% for steps. Intra-RR showed an accuracy of 98.4 ± 1.1% for phases and 88.1 ± 8.1% for steps.

CONCLUSION

The proposed ontology shows an excellent inter- and intra-RR and should therefore be implemented routinely in phase and step annotation of LRYGB.

Virtual reality environments to train soft skills in medical and nursing education: a technical feasibility study between France and Japan

PURPOSE

To meet the urgent and massive training needs of healthcare professionals, the use of digital technologies is proving increasingly relevant, and the rise of digital training platforms shows their usefulness and possibilities. However, despite the impact of these platforms on the medical skills learning, cultural differences are rarely factored in the implementation of these training environments.

METHODS

By using the Scrub Nurse Non-Technical Skills Training System (SunSet), we developed a methodology enabling the adaptation of a virtual reality-based environment and scenarios from French to Japanese cultural and medical practices. We then conducted a technical feasibility study between France and Japan to assess virtual reality simulations acceptance among scrub nurses.

RESULTS

Results in term of acceptance do not reveal major disparity between both populations, and the only emerging significant difference between both groups is on the Behavioral Intention, which is significantly higher for the French scrub nurses. In both cases, participants had a positive outlook.

CONCLUSION

The findings suggest that the methodology we have implemented can be further used in the context of cultural adaptation of non-technical skills learning scenarios in virtual environments for the training and assessment of health care personnel.

Logic programming for deliberative robotic task planning

Over the last decade, the use of robots in production and daily life has increased. With increasingly complex tasks and interaction in different environments including humans, robots are required a higher level of autonomy for efficient deliberation. Task planning is a key element of deliberation. It combines elementary operations into a structured plan to satisfy a prescribed goal, given specifications on the robot and the environment. In this manuscript, we present a survey on recent advances in the application of logic programming to the problem of task planning. Logic programming offers several advantages compared to other approaches, including greater expressivity and interpretability which may aid in the development of safe and reliable robots. We analyze different planners and their suitability for specific robotic applications, based on expressivity in domain representation, computational efficiency and software implementation. In this way, we support the robotic designer in choosing the best tool for his application.

Surgical declarative knowledge learning: concept and acceptability study

Improving surgical training by means of technology assistance is an important challenge that aims to directly impact surgical quality. Surgical training includes the acquisition of two categories of knowledge: declarative knowledge (i.e. 'knowing what') and procedural knowledge (i.e. 'knowing how'). It is essential to acquire both before performing any particular surgery. There are currently many tools for acquiring procedural knowledge, such as simulators. However, few approaches or tools allow a trainer to formalize and record surgical declarative knowledge, and a trainee to have easy access to it. In this paper, we propose an approach for structuring surgical declarative knowledge according to procedural knowledge and based on surgical process modeling. A dedicated software application has been implemented. We evaluated the concept and the software usability on two procedures with different medical populations: endoscopic third ventriculostomy involving 6 neurosurgeons and preparation of a surgical table for craniotomy involving 4 scrub nurses. The results of both studies show that surgical process models could be a well-adapted approach for structuring and visualizing surgical declarative knowledge. The software application was perceived by neurosurgeons and scrub nurses as an innovative tool for managing and presenting surgical knowledge. The preliminary results show that the feasibility of the proposed approach and the acceptability and usability of the corresponding software. Future experiments will study impact of such an approach on knowledge acquisition.

Surgomics: personalized prediction of morbidity, mortality and long-term outcome in surgery using machine learning on multimodal data

BACKGROUND

Personalized medicine requires the integration and analysis of vast amounts of patient data to realize individualized care. With Surgomics, we aim to facilitate personalized therapy recommendations in surgery by integration of intraoperative surgical data and their analysis with machine learning methods to leverage the potential of this data in analogy to Radiomics and Genomics.

METHODS

We defined Surgomics as the entirety of surgomic features that are process characteristics of a surgical procedure automatically derived from multimodal intraoperative data to quantify processes in the operating room. In a multidisciplinary team we discussed potential data sources like endoscopic videos, vital sign monitoring, medical devices and instruments and respective surgomic features. Subsequently, an online questionnaire was sent to experts from surgery and (computer) science at multiple centers for rating the features' clinical relevance and technical feasibility.

RESULTS

In total, 52 surgomic features were identified and assigned to eight feature categories. Based on the expert survey (n = 66 participants) the feature category with the highest clinical relevance as rated by surgeons was "surgical skill and quality of performance" for morbidity and mortality (9.0 ± 1.3 on a numerical rating scale from 1 to 10) as well as for long-term (oncological) outcome (8.2 ± 1.8). The feature category with the highest feasibility to be automatically extracted as rated by (computer) scientists was "Instrument" (8.5 ± 1.7). Among the surgomic features ranked as most relevant in their respective category were "intraoperative adverse events", "action performed with instruments", "vital sign monitoring", and "difficulty of surgery".

CONCLUSION

Surgomics is a promising concept for the analysis of intraoperative data. Surgomics may be used together with preoperative features from clinical data and Radiomics to predict postoperative morbidity, mortality and long-term outcome, as well as to provide tailored feedback for surgeons.

Service-oriented Device Connectivity interface for a situation recognition system in the OR

Purpose

Context awareness in the operating room (OR) is important to realize targeted assistance to support actors during surgery. A situation recognition system (SRS) is used to interpret intraoperative events and derive an intraoperative situation from these. To achieve a modular system architecture, it is desirable to de-couple the SRS from other system components. This leads to the need of an interface between such an SRS and context-aware systems (CAS). This work aims to provide an open standardized interface to enable loose coupling of the SRS with varying CAS to allow vendor-independent device orchestrations.

Methods

A requirements analysis investigated limiting factors that currently prevent the integration of CAS in today's ORs. These elicited requirements enabled the selection of a suitable base architecture. We examined how to specify this architecture with the constraints of an interoperability standard. The resulting middleware was integrated into a prototypic SRS and our system for intraoperative support, the OR-Pad, as exemplary CAS for evaluating whether our solution can enable context-aware assistance during simulated orthopedical interventions.

Results

The emerging Service-oriented Device Connectivity (SDC) standard series was selected to specify and implement a middleware for providing the interpreted contextual information while the SRS and CAS are loosely coupled. The results were verified within a proof of concept study using the OR-Pad demonstration scenario. The fulfillment of the CAS’ requirements to act context-aware, conformity to the SDC standard series, and the effort for integrating the middleware in individual systems were evaluated. The semantically unambiguous encoding of contextual information depends on the further standardization process of the SDC nomenclature. The discussion of the validity of these results proved the applicability and transferability of the middleware.

Conclusion

The specified and implemented SDC-based middleware shows the feasibility of loose coupling an SRS with unknown CAS to realize context-aware assistance in the OR.

Surgical data science - from concepts toward clinical translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Surgical Process Modeling for Open Spinal Surgeries

Modern operating rooms are becoming increasingly advanced thanks to the emerging medical technologies and cutting-edge surgical techniques. Current surgeries are transitioning into complex processes that involve information and actions from multiple resources. When designing context-aware medical technologies for a given intervention, it is of utmost importance to have a deep understanding of the underlying surgical process. This is essential to develop technologies that can correctly address the clinical needs and can adapt to the existing workflow. Surgical Process Modeling (SPM) is a relatively recent discipline that focuses on achieving a profound understanding of the surgical workflow and providing a model that explains the elements of a given surgery as well as their sequence and hierarchy, both in quantitative and qualitative manner. To date, a significant body of work has been dedicated to the development of comprehensive SPMs for minimally invasive baroscopic and endoscopic surgeries, while such models are missing for open spinal surgeries. In this paper, we provide SPMs common open spinal interventions in orthopedics. Direct video observations of surgeries conducted in our institution were used to derive temporal and transitional information about the surgical activities. This information was later used to develop detailed SPMs that modeled different primary surgical steps and highlighted the frequency of transitions between the surgical activities made within each step. Given the recent emersion of advanced techniques that are tailored to open spinal surgeries (e.g., artificial intelligence methods for intraoperative guidance and navigation), we believe that the SPMs provided in this study can serve as the basis for further advancement of next-generation algorithms dedicated to open spinal interventions that require a profound understanding of the surgical workflow (e.g., automatic surgical activity recognition and surgical skill evaluation). Furthermore, the models provided in this study can potentially benefit the clinical community through standardization of the surgery, which is essential for surgical training.

A Probabilistic Approach To Surgical Tasks and Skill Metrics

Identifying and quantifying the activities that compose surgery is essential for effective interventions, computer-aided analyses and the advancement of surgical data science. For example, recent studies have shown that objective metrics (referred to as objective performance indicators, OPIs) computed during key surgical tasks correlate with surgeon skill and clinical outcomes. Unambiguous identification of these surgical tasks can be particularly challenging for both human annotators and algorithms. Each surgical procedure has multiple approaches, each surgeon has their own level of skill, and the initiation and termination of surgical tasks can be subject to interpretation. As such, human annotators and machine learning models face the same basic problem, accurately identifying the boundaries of surgical tasks despite variable and unstructured information. For use in surgeon feedback, OPIs should also be robust to the variability and diversity in this data. To mitigate this difficulty, we propose a probabilistic approach to surgical task identification and calculation of OPIs. Rather than relying on tasks that are identified by hard temporal boundaries, we demonstrate an approach that relies on distributions of start and stop times, for a probabilistic interpretation of when the task was performed. We first use hypothetical data to outline how this approach is superior to other conventional approaches. Then we present similar analyses on surgical data. We find that when surgical tasks are identified by their individual probabilities, the resulting OPIs are less sensitive to noise in the identification of the start and stop times. These results suggest that this probabilistic approach holds promise for the future of surgical data science.

An Overview of Ontologies in Virtual Reality-Based Training for Healthcare Domain

Virtual reality (VR) is one of the state-of-the-art technological applications in the healthcare domain. One major aspect of VR applications in this domain includes virtual reality-based training (VRT), which simplifies the complicated visualization process of diagnosis, treatment, disease analysis, and prevention. However, not much is known on how well the domain knowledge is shared and considered in the development of VRT applications. A pertinent mechanism, known as ontology, has acted as an enabler toward making the domain knowledge more explicit. Hence, this paper presents an overview to reveal the basic concepts and explores the extent to which ontologies are used in VRT development for medical education and training in the healthcare domain. From this overview, a base of knowledge for VRT development is proposed to initiate a comprehensive strategy in creating an effective ontology design for VRT applications in the healthcare domain.

SAGES consensus recommendations on an annotation framework for surgical video

BACKGROUND

The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration.

METHODS

Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups.

RESULTS

After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established.

CONCLUSIONS

While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Ontology-Based Knowledge Representation in Robotic Systems: A Survey Oriented toward Applications

Knowledge representation in autonomous robots with social roles has steadily gained importance through their supportive task assistance in domestic, hospital, and industrial activities. For active assistance, these robots must process semantic knowledge to perform the task more efficiently. In this context, ontology-based knowledge representation and reasoning (KR & R) techniques appear as a powerful tool and provide sophisticated domain knowledge for processing complex robotic tasks in a real-world environment. In this article, we surveyed ontology-based semantic representation unified into the current state of robotic knowledge base systems, with our aim being three-fold: (i) to present the recent developments in ontology-based knowledge representation systems that have led to the effective solutions of real-world robotic applications; (ii) to review the selected knowledge-based systems in seven dimensions: application, idea, development tools, architecture, ontology scope, reasoning scope, and limitations; (iii) to pin-down lessons learned from the review of existing knowledge-based systems for designing better solutions and delineating research limitations that might be addressed in future studies. This survey article concludes with a discussion of future research challenges that can serve as a guide to those who are interested in working on the ontology-based semantic knowledge representation systems for autonomous robots.

Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery

BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome's point of view is clearly indicated by the available surgical error statistics.

Artificial Intelligence-Assisted Surgery: Potential and Challenges

BACKGROUND

Artificial intelligence (AI) has recently achieved considerable success in different domains including medical applications. Although current advances are expected to impact surgery, up until now AI has not been able to leverage its full potential due to several challenges that are specific to that field.

SUMMARY

This review summarizes data-driven methods and technologies needed as a prerequisite for different AI-based assistance functions in the operating room. Potential effects of AI usage in surgery will be highlighted, concluding with ongoing challenges to enabling AI for surgery.

KEY MESSAGES

AI-assisted surgery will enable data-driven decision-making via decision support systems and cognitive robotic assistance. The use of AI for workflow analysis will help provide appropriate assistance in the right context. The requirements for such assistance must be defined by surgeons in close cooperation with computer scientists and engineers. Once the existing challenges will have been solved, AI assistance has the potential to improve patient care by supporting the surgeon without replacing him or her.

Data Tagging in Medical Images: A Survey of the State-of-Art

A huge amount of medical data is generated every second, and a significant percentage of the data are images that need to be analyzed and processed. One of the key challenges in this regard is the recovery of the data of medical images. The medical image recovery procedure should be done automatically by the computers that are the method of identifying object concepts and assigning homologous tags to them. To discover the hidden concepts in the medical images, the lowlevel characteristics should be used to achieve high-level concepts and that is a challenging task. In any specific case, it requires human involvement to determine the significance of the image. To allow machine-based reasoning on the medical evidence collected, the data must be accompanied by additional interpretive semantics; a change from a pure data-intensive methodology to a model of evidence rich in semantics. In this state-of-art, data tagging methods related to medical images are surveyed which is an important aspect for the recognition of a huge number of medical images. Different types of tags related to the medical image, prerequisites of medical data tagging, different techniques to develop medical image tags, different medical image tagging algorithms and different tools that are used to create the tags are discussed in this paper. The aim of this state-of-art paper is to produce a summary and a set of guidelines for using the tags for the identification of medical images and to identify the challenges and future research directions of tagging medical images.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.

Learning procedural skills with a virtual reality simulator: An acceptability study

BACKGROUND

Virtual Reality (VR) simulation has recently been developed and has improved surgical training. Most VR simulators focus on learning technical skills and few on procedural skills. Studies that evaluated VR simulators focused on feasibility, reliability or easiness of use, but few of them used a specific acceptability measurement tool.

OBJECTIVES

The aim of the study was to assess acceptability and usability of a new VR simulator for procedural skill training among scrub nurses, based on the Unified Theory of Acceptance and Use of Technology (UTAUT) model.

PARTICIPANTS

The simulator training system was tested with a convenience sample of 16 non-expert users and 13 expert scrub nurses from the neurosurgery department of a French University Hospital.

METHODS

The scenario was designed to train scrub nurses in the preparation of the instrumentation table for a craniotomy in the operating room (OR).

RESULTS

Acceptability of the VR simulator was demonstrated with no significant difference between expert scrub nurses and non-experts. There was no effect of age, gender or expertise. Workload, immersion and simulator sickness were also rated equally by all participants. Most participants stressed its pedagogical interest, fun and realism, but some of them also regretted its lack of visual comfort.

CONCLUSION

This VR simulator designed to teach surgical procedures can be widely used as a tool in initial or vocational training.

Surgical process modelling strategies: which method to choose for determining workflow?

The vital role of surgeries in healthcare requires a constant attention to improvement. Surgical process modelling is an innovative and rather recently introduced approach for tackling the issues in today's complex surgeries. This modelling field is very challenging and still under development, therefore, it is not always clear which modelling strategy would best fit the needs in which situations. The aim of this study was to provide a guide for matching the choice of modelling strategies for determining surgical workflows. In this work, the concepts associated with surgical process modelling are described, aiming to clarify them and to promote their use in future studies. The relationship of these concepts and the possible combinations of the suitable approaches for modelling strategies are elaborated and the criteria for opting for the proper modelling strategy are discussed.