SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education)

BACKGROUND

Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various factors, including data structure, acquisition, storage, and sharing; data use and exploration, and finally data governance, which encompasses all ethical and legal regulations associated with the data. There is a universal need among stakeholders in surgical data science to establish standardized frameworks that address all aspects of this lifecycle to ensure data quality and purpose.

METHODS

Working groups were formed, among 48 representatives from academia and industry, including clinicians, computer scientists and industry representatives. These working groups focused on: Data Use, Data Structure, Data Exploration, and Data Governance. After working group and panel discussions, a modified Delphi process was conducted.

RESULTS

The resulting Delphi consensus provides conceptualized and structured recommendations for each domain related to surgical video data. We identified the key stakeholders within the data lifecycle and formulated comprehensive, easily understandable, and widely applicable guidelines for data utilization. Standardization of data structure should encompass format and quality, data sources, documentation, metadata, and account for biases within the data. To foster scientific data exploration, datasets should reflect diversity and remain adaptable to future applications. Data governance must be transparent to all stakeholders, addressing legal and ethical considerations surrounding the data.

CONCLUSION

This consensus presents essential recommendations around the generation of standardized and diverse surgical video databanks, accounting for multiple stakeholders involved in data generation and use throughout its lifecycle. Following the SAGES annotation framework, we lay the foundation for standardization of data use, structure, and exploration. A detailed exploration of requirements for adequate data governance will follow.

Surgical data science - from concepts toward clinical translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

A Platform and Multisided Market for Translational, Software-Defined Medical Procedures in the Operating Room (OP 4.1): Proof-of-Concept Study

Background

Although digital and data-based technologies are widespread in various industries in the context of Industry 4.0, the use of smart connected devices in health care is still in its infancy. Innovative solutions for the medical environment are affected by difficult access to medical device data and high barriers to market entry because of proprietary systems.

Objective

In the proof-of-concept project OP 4.1, we show the business viability of connecting and augmenting medical devices and data through software add-ons by giving companies a technical and commercial platform for the development, implementation, distribution, and billing of innovative software solutions.

Methods

The creation of a central platform prototype requires the collaboration of several independent market contenders, including medical users, software developers, medical device manufacturers, and platform providers. A dedicated consortium of clinical and scientific partners as well as industry partners was set up.

Results

We demonstrate the successful development of the prototype of a user-centric, open, and extensible platform for the intelligent support of processes starting with the operating room. By connecting heterogeneous data sources and medical devices from different manufacturers and making them accessible for software developers and medical users, the cloud-based platform OP 4.1 enables the augmentation of medical devices and procedures through software-based solutions. The platform also allows for the demand-oriented billing of apps and medical devices, thus permitting software-based solutions to fast-track their economic development and become commercially successful.

Conclusions

The technology and business platform OP 4.1 creates a multisided market for the successful development, implementation, distribution, and billing of new software solutions in the operating room and in the health care sector in general. Consequently, software-based medical innovation can be translated into clinical routine quickly, efficiently, and cost-effectively, optimizing the treatment of patients through smartly assisted procedures.

[Surgery 4.0-are we ready?]

The increasing networking of data systems in medicine is not only leading to modern interdisciplinarity in the sense of cooperation between different medical departments, but also poses new challenges regarding the building and room infrastructure. The surgical operating room of the future expands or augments its reality, away from the pure building characteristics, towards an intelligent and communicative space platform. The building infrastructure (operating theatre) serves as sensor and actuator. Thus, it is possible to inform about missing diagnostics as well as to register them directly in the contextualization of the planned surgical intervention or to integrate them into the processes. Integrated operating theatres represent a comprehensive computer platform based on a corresponding system architecture with software-based protocols. An underlying modular system consisting of various modules for image acquisition and analysis, interaction and visualization supports the integration and merging of heterogeneous data that are generated in a hospital operation. Integral building data (e.g., air conditioning, lighting control, device registration) are merged with patient-related data (age, type of illness, concomitant diseases, existing diagnostic CT and MRI images). New systems coming onto the market, as well as already existing systems will have to be measured by the extent to which they will be able to guarantee this integration of information-similar to the development from mobile phone to smartphone. Cost reduction should not be the only legitimizing argument for the market launch, but the vision of a new quality of surgical perception and action.

Why rankings of biomedical image analysis competitions should be interpreted with care

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Large-scale medical image annotation with crowd-powered algorithms

Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.

Toward a standard ontology of surgical process models

PURPOSE

The development of common ontologies has recently been identified as one of the key challenges in the emerging field of surgical data science (SDS). However, past and existing initiatives in the domain of surgery have mainly been focussing on individual groups and failed to achieve widespread international acceptance by the research community. To address this challenge, the authors of this paper launched a European initiative-OntoSPM Collaborative Action-with the goal of establishing a framework for joint development of ontologies in the field of SDS. This manuscript summarizes the goals and the current status of the international initiative.

METHODS

A workshop was organized in 2016, gathering the main European research groups having experience in developing and using ontologies in this domain. It led to the conclusion that a common ontology for surgical process models (SPM) was absolutely needed, and that the existing OntoSPM ontology could provide a good starting point toward the collaborative design and promotion of common, standard ontologies on SPM.

RESULTS

The workshop led to the OntoSPM Collaborative Action-launched in mid-2016-with the objective to develop, maintain and promote the use of common ontologies of SPM relevant to the whole domain of SDS. The fundamental concept, the architecture, the management and curation of the common ontology have been established, making it ready for wider public use.

CONCLUSION

The OntoSPM Collaborative Action has been in operation for 24 months, with a growing dedicated membership. Its main result is a modular ontology, undergoing constant updates and extensions, based on the experts' suggestions. It remains an open collaborative action, which always welcomes new contributors and applications.

Toward knowledge-based liver surgery: holistic information processing for surgical decision support

PURPOSE

Malignant neoplasms of the liver are among the most frequent cancers worldwide. Given the diversity of options for liver cancer therapy, the choice of treatment depends on various parameters including patient condition, tumor size and location, liver function, and previous interventions. To address this issue, we present the first approach to treatment strategy planning based on holistic processing of patient-individual data, practical knowledge (i.e., case knowledge), and factual knowledge (e.g., clinical guidelines and studies).

METHODS

The contributions of this paper are as follows: (1) a formalized dynamic patient model that incorporates all the heterogeneous data acquired for a specific patient in the whole course of disease treatment; (2) a concept for formalizing factual knowledge; and (3) a technical infrastructure that enables storing, accessing, and processing of heterogeneous data to support clinical decision making.

RESULTS

Our patient model, which currently covers 602 patient-individual parameters, was successfully instantiated for 184 patients. It was sufficiently comprehensive to serve as the basis for the formalization of a total of 72 rules extracted from studies on patients with colorectal liver metastases or hepatocellular carcinoma. For a subset of 70 patients with these diagnoses, the system derived an average of [Formula: see text] assertions per patient.

CONCLUSION

The proposed concept paves the way for holistic treatment strategy planning by enabling joint storing and processing of heterogeneous data from various information sources.