Investigating the Ethical and Data Governance Issues of Artificial Intelligence in Surgery: Protocol for a Delphi Study

DATA GOVERNANCE in digital surgery

Technological and computer advances have led to a "new era" of Surgery called Digital Surgery. In it, the management of information is the key. The development of Artificial Intelligence requires "Big Data" to create its algorithms. The use of digital technology for the systematic capture of data from the surgical process raises ethical issues of privacy, property, and consent. The use of these out-of-control data creates uncertainty and can be a source of mistrust and refusal by surgeons to allow its use, requiring a framework for the correct management of them. This paper exposes the current situation of Data Governance in Digital Surgery, the challenges posed and the lines of action necessary to resolve the areas of uncertainty that have arisen in the process, in which the surgeon must play a relevant role.

European Robotic Surgery Consensus (ERSC): Protocol for the development of a consensus in robotic training for gastrointestinal surgery trainees

BACKGROUND

The rapid adoption of robotic surgical systems across Europe has led to a critical gap in training and credentialing for gastrointestinal (GI) surgeons. Currently, there is no existing standardised curriculum to guide robotic training, assessment and certification for GI trainees. This manuscript describes the protocol to achieve a pan-European consensus on the essential components of a comprehensive training programme for GI robotic surgery through a five-stage process.

METHODS AND ANALYSIS

In Stage 1, a Steering Committee, consisting of international experts, trainees and educationalists, has been established to lead and coordinate the consensus development process. In Stage 2, a systematic review of existing multi-specialty robotic training curricula will be performed to inform the formulation of key position statements. In Stage 3, a comprehensive survey will be disseminated across Europe to capture the current state of robotic training and identify potential challenges and opportunities for improvement. In Stage 4, an international panel of GI surgeons, trainees, and robotic theatre staff will participate in a three-round Delphi process, seeking ≥ 70% agreement on crucial aspects of the training curriculum. Industry and patient representatives will be involved as external advisors throughout this process. In Stage 5, the robotic training curriculum for GI trainees will be finalised in a dedicated consensus meeting, culminating in the production of an Explanation and Elaboration (E&E) document.

REGISTRATION DETAILS

The study protocol has been registered on the Open Science Framework (https://osf.io/br87d/).

Potential of Large Language Models in Health Care: Delphi Study

BACKGROUND

A large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications.

OBJECTIVE

The aim of this adapted Delphi study was to collect researchers' opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care.

METHODS

We invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items.

RESULTS

The first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data.

CONCLUSIONS

Future research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.

International perspectives on measuring national digital public health system maturity through a multidisciplinary Delphi study

Unlocking the full potential of digital public health (DiPH) systems requires a comprehensive tool to assess their maturity. While the World Health Organization and the International Telecommunication Union released a toolkit in 2012 covering various aspects of digitalizing national healthcare systems, a holistic maturity assessment tool has been lacking ever since. To bridge this gap, we conducted a pioneering Delphi study, to which 54 experts from diverse continents and academic fields actively contributed to at least one of three rounds. 54 experts participated in developing and rating multidisciplinary quality indicators to measure the maturity of national digital public health systems. Participants established consensus on these indicators with a threshold of 70% agreement on indicator importance. Eventually, 96 indicators were identified and agreed upon by experts. Notably, 48% of these indicators were found to align with existing validated tools, highlighting their relevance and reliability. However, further investigation is required to assess the suitability and applicability of all the suggestions put forward by our participants. Nevertheless, this Delphi study is an essential initial stride toward a comprehensive measurement tool for DiPH system maturity. By working towards a standardized assessment of DiPH system maturity, we aim to empower decision-makers to make informed choices, optimize resource allocation, and drive innovation in healthcare delivery. The results of this study mark a significant milestone in advancing DiPH on a global scale.

Investigating the national implementation of SMS and mobile messaging in population screening (The SIPS study)

BACKGROUND

The increasing use of mobile messaging within healthcare, poses challenges for screening programmes, which involve communicating with large, diverse populations. This modified Delphi study aimed to create guidance regarding the use of mobile messaging for screening programmes, to facilitate greater, and equitable screening uptake.

METHODS

Initial recommendations were derived from a literature review, expert scoping questionnaire, public consultation, and discussion with relevant national organisations. Experts from the fields of public health, screening commissioning, industry and academia voted upon the importance and feasibility of these recommendations across two consensus rounds, using a 5-point Likert scale. Items reaching consensus, defined a priori at 70%, on importance and feasibility formed 'core' recommendations. Those reaching this threshold on importance only, were labelled 'desirable'. All items were subsequently discussed at an expert meeting to confirm suitability.

FINDINGS

Of the initial 101 items, 23 reached consensus regarding importance and feasibility. These 'core' items were divided across six domains: message content, timing, delivery, evaluation, security, and research considerations. 'Core' items such as explicitly specifying the sender and the role of patient involvement in development of screening message research had the highest agreement. A further 17 'desirable' items reached consensus regarding importance, but not feasibility, including the integration into GP services to enable telephone verification.

INTERPRETATION

These findings forming national guidance for services, will enable programmes to overcome implementation challenges and facilitate uptake of screening invitations. By providing a list of desirable items, this study provides areas for future consideration, as technological innovation in messaging continues to grow.

FUNDING

NIHR Imperial Patient Safety Translational Research Centre.

Identifying essential factors that influence user engagement with digital mental health tools in clinical care settings: Protocol for a Delphi study

Introduction

Improving effective user engagement with digital mental health tools has become a priority in enabling the value of digital health. With increased interest from the mental health community in embedding digital health tools as part of care delivery, there is a need to examine and identify the essential factors in influencing user engagement with digital mental health tools in clinical care. The current study will use a Delphi approach to gain consensus from individuals with relevant experience and expertise (e.g. patients, clinicians and healthcare administrators) on factors that influence user engagement (i.e. an essential factor).

Methods

Participants will be invited to complete up to four rounds of online surveys. The first round of the Delphi study comprises of reviewing existing factors identified in literature and commenting on whether any factors they believe are important are missing from the list. Subsequent rounds will involve asking participants to rate the perceived impact of each factor in influencing user engagement with digital mental health tools in clinical care contexts. This work is expected to consolidate the perspectives from relevant stakeholders and the academic literature to identify a core set of factors considered essential in influencing user engagement with digital mental health tools in clinical care contexts.

A Delphi consensus statement for digital surgery

The use of digital technology is increasing rapidly across surgical specialities, yet there is no consensus for the term ‘digital surgery’. This is critical as digital health technologies present technical, governance, and legal challenges which are unique to the surgeon and surgical patient. We aim to define the term digital surgery and the ethical issues surrounding its clinical application, and to identify barriers and research goals for future practice. 38 international experts, across the fields of surgery, AI, industry, law, ethics and policy, participated in a four-round Delphi exercise. Issues were generated by an expert panel and public panel through a scoping questionnaire around key themes identified from the literature and voted upon in two subsequent questionnaire rounds. Consensus was defined if >70% of the panel deemed the statement important and <30% unimportant. A final online meeting was held to discuss consensus statements. The definition of digital surgery as the use of technology for the enhancement of preoperative planning, surgical performance, therapeutic support, or training, to improve outcomes and reduce harm achieved 100% consensus agreement. We highlight key ethical issues concerning data, privacy, confidentiality and public trust, consent, law, litigation and liability, and commercial partnerships within digital surgery and identify barriers and research goals for future practice. Developers and users of digital surgery must not only have an awareness of the ethical issues surrounding digital applications in healthcare, but also the ethical considerations unique to digital surgery. Future research into these issues must involve all digital surgery stakeholders including patients.

Machine learning for technical skill assessment in surgery: a systematic review

Accurate and objective performance assessment is essential for both trainees and certified surgeons. However, existing methods can be time consuming, labor intensive, and subject to bias. Machine learning (ML) has the potential to provide rapid, automated, and reproducible feedback without the need for expert reviewers. We aimed to systematically review the literature and determine the ML techniques used for technical surgical skill assessment and identify challenges and barriers in the field. A systematic literature search, in accordance with the PRISMA statement, was performed to identify studies detailing the use of ML for technical skill assessment in surgery. Of the 1896 studies that were retrieved, 66 studies were included. The most common ML methods used were Hidden Markov Models (HMM, 14/66), Support Vector Machines (SVM, 17/66), and Artificial Neural Networks (ANN, 17/66). 40/66 studies used kinematic data, 19/66 used video or image data, and 7/66 used both. Studies assessed the performance of benchtop tasks (48/66), simulator tasks (10/66), and real-life surgery (8/66). Accuracy rates of over 80% were achieved, although tasks and participants varied between studies. Barriers to progress in the field included a focus on basic tasks, lack of standardization between studies, and lack of datasets. ML has the potential to produce accurate and objective surgical skill assessment through the use of methods including HMM, SVM, and ANN. Future ML-based assessment tools should move beyond the assessment of basic tasks and towards real-life surgery and provide interpretable feedback with clinical value for the surgeon.PROSPERO: CRD42020226071.

Implementation Frameworks for Artificial Intelligence Translation Into Health Care Practice: Scoping Review

Background

Significant efforts have been made to develop artificial intelligence (AI) solutions for health care improvement. Despite the enthusiasm, health care professionals still struggle to implement AI in their daily practice.

Objective

This paper aims to identify the implementation frameworks used to understand the application of AI in health care practice.

Methods

A scoping review was conducted using the Cochrane, Evidence Based Medicine Reviews, Embase, MEDLINE, and PsycINFO databases to identify publications that reported frameworks, models, and theories concerning AI implementation in health care. This review focused on studies published in English and investigating AI implementation in health care since 2000. A total of 2541 unique publications were retrieved from the databases and screened on titles and abstracts by 2 independent reviewers. Selected articles were thematically analyzed against the Nilsen taxonomy of implementation frameworks, and the Greenhalgh framework for the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) of health care technologies.

Results

In total, 7 articles met all eligibility criteria for inclusion in the review, and 2 articles included formal frameworks that directly addressed AI implementation, whereas the other articles provided limited descriptions of elements influencing implementation. Collectively, the 7 articles identified elements that aligned with all the NASSS domains, but no single article comprehensively considered the factors known to influence technology implementation. New domains were identified, including dependency on data input and existing processes, shared decision-making, the role of human oversight, and ethics of population impact and inequality, suggesting that existing frameworks do not fully consider the unique needs of AI implementation.

Conclusions

This literature review demonstrates that understanding how to implement AI in health care practice is still in its early stages of development. Our findings suggest that further research is needed to provide the knowledge necessary to develop implementation frameworks to guide the future implementation of AI in clinical practice and highlight the opportunity to draw on existing knowledge from the field of implementation science.

Investigating the Implementation of SMS and Mobile Messaging in Population Screening (the SIPS Study): Protocol for a Delphi Study

BACKGROUND

The use of mobile messaging, including SMS, and web-based messaging in health care has grown significantly. Using messaging to facilitate patient communication has been advocated in several circumstances, including population screening. These programs, however, pose unique challenges to mobile communication, as messaging is often sent from a central hub to a diverse population with differing needs. Despite this, there is a paucity of robust frameworks to guide implementation.

OBJECTIVE

The aim of this protocol is to describe the methods that will be used to develop a guide for the principles of use of mobile messaging for population screening programs in England.

METHODS

This modified Delphi study will be conducted in two parts: evidence synthesis and consensus generation. The former will include a review of literature published from January 1, 2000, to October 1, 2021. This will elicit key themes to inform an online scoping questionnaire posed to a group of experts from academia, clinical medicine, industry, and public health. Thematic analysis of free-text responses by two independent authors will elicit items to be used during consensus generation. Patient and Public Involvement and Engagement groups will be convened to ensure that a comprehensive item list is generated that represents the public's perspective. Each item will then be anonymously voted on by experts as to its importance and feasibility of implementation in screening during three rounds of a Delphi process. Consensus will be defined a priori at 70%, with items considered important and feasible being eligible for inclusion in the final recommendation. A list of desirable items (ie, important but not currently feasible) will be developed to guide future work.

RESULTS

The Institutional Review Board at Imperial College London has granted ethical approval for this study (reference 20IC6088). Results are expected to involve a list of recommendations to screening services, with findings being made available to screening services through Public Health England. This study will, thus, provide a formal guideline for the use of mobile messaging in screening services and will provide future directions in this field.

CONCLUSIONS

The use of mobile messaging has grown significantly across health care services, especially given the COVID-19 pandemic, but its implementation in screening programs remains challenging. This modified Delphi approach with leading experts will provide invaluable insights into facilitating the incorporation of messaging into these programs and will create awareness of future developments in this area.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/32660.