Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Glucocorticoid therapy for sepsis in the AI era: a survey on current and future approaches

Sepsis, a life-threatening medical condition, manifests as new or worsening organ failures due to a dysregulated host response to infection. Many patients with sepsis have manifested a hyperinflammatory phenotype leading to the identification of inflammatory modulation by corticosteroids as a key treatment modality. However, the optimal use of corticosteroids in sepsis treatment remains a contentious subject, necessitating a deeper understanding of their physiological and pharmacological effects. Our study conducts a comprehensive review of randomized controlled trials (RCTs) focusing on traditional corticosteroid treatment in sepsis, alongside an analysis of evolving clinical guidelines. Additionally, we explore the emerging role of artificial intelligence (AI) in medicine, particularly in diagnosing, prognosticating, and treating sepsis. AI's advanced data processing capabilities reveal new avenues for enhancing corticosteroid therapeutic strategies in sepsis. The integration of AI in sepsis treatment has the potential to address existing gaps in knowledge, especially in the application of corticosteroids. Our findings suggest that combining corticosteroid therapy with AI-driven insights could lead to more personalized and effective sepsis treatments. This approach holds promise for improving clinical outcomes and presents a significant advancement in the management of this complex and often fatal condition.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Toward a responsible future: recommendations for AI-enabled clinical decision support

BACKGROUND

Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging.

OBJECTIVES

This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients.

MATERIALS AND METHODS

In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process.

RESULTS

Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided.

DISCUSSION

AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow.

CONCLUSIONS

Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

Machine Learning-Based Predictive Models for Patients with Venous Thromboembolism: A Systematic Review

BACKGROUND

 Venous thromboembolism (VTE) is a chronic disorder with a significant health and economic burden. Several VTE-specific clinical prediction models (CPMs) have been used to assist physicians in decision-making but have several limitations. This systematic review explores if machine learning (ML) can enhance CPMs by analyzing extensive patient data derived from electronic health records. We aimed to explore ML-CPMs' applications in VTE for risk stratification, outcome prediction, diagnosis, and treatment.

METHODS

 Three databases were searched: PubMed, Google Scholar, and IEEE electronic library. Inclusion criteria focused on studies using structured data, excluding non-English publications, studies on non-humans, and certain data types such as natural language processing and image processing. Studies involving pregnant women, cancer patients, and children were also excluded. After excluding irrelevant studies, a total of 77 studies were included.

RESULTS

 Most studies report that ML-CPMs outperformed traditional CPMs in terms of receiver operating area under the curve in the four clinical domains that were explored. However, the majority of the studies were retrospective, monocentric, and lacked detailed model architecture description and external validation, which are essential for quality audit. This review identified research gaps and highlighted challenges related to standardized reporting, reproducibility, and model comparison.

CONCLUSION

 ML-CPMs show promise in improving risk assessment and individualized treatment recommendations in VTE. Apparently, there is an urgent need for standardized reporting and methodology for ML models, external validation, prospective and real-world data studies, as well as interventional studies to evaluate the impact of artificial intelligence in VTE.

Studies of Artificial Intelligence/Machine Learning Registered on ClinicalTrials.gov: Cross-Sectional Study With Temporal Trends, 2010-2023

BACKGROUND

The rapid growth of research in artificial intelligence (AI) and machine learning (ML) continues. However, it is unclear whether this growth reflects an increase in desirable study attributes or merely perpetuates the same issues previously raised in the literature.

OBJECTIVE

This study aims to evaluate temporal trends in AI/ML studies over time and identify variations that are not apparent from aggregated totals at a single point in time.

METHODS

We identified AI/ML studies registered on ClinicalTrials.gov with start dates between January 1, 2010, and December 31, 2023. Studies were included if AI/ML-specific terms appeared in the official title, detailed description, brief summary, intervention, primary outcome, or sponsors' keywords. Studies registered as systematic reviews and meta-analyses were excluded. We reported trends in AI/ML studies over time, along with study characteristics that were fast-growing and those that remained unchanged during 2010-2023.

RESULTS

Of 3106 AI/ML studies, only 7.6% (n=235) were regulated by the US Food and Drug Administration. The most common study characteristics were randomized (56.2%; 670/1193; interventional) and prospective (58.9%; 1126/1913; observational) designs; a focus on diagnosis (28.2%; 335/1190) and treatment (24.4%; 290/1190); hospital/clinic (44.2%; 1373/3106) or academic (28%; 869/3106) sponsorship; and neoplasm (12.9%; 420/3245), nervous system (12.2%; 395/3245), cardiovascular (11.1%; 356/3245) or pathological conditions (10%; 325/3245; multiple counts per study possible). Enrollment data were skewed to the right: maximum 13,977,257; mean 16,962 (SD 288,155); median 255 (IQR 80-1000). The most common size category was 101-1000 (44.8%; 1372/3061; excluding withdrawn or missing), but large studies (n>1000) represented 24.1% (738/3061) of all studies: 29% (551/1898) of observational studies and 16.1% (187/1163) of trials. Study locations were predominantly in high-income countries (75.3%; 2340/3106), followed by upper-middle-income (21.7%; 675/3106), lower-middle-income (2.8%; 88/3106), and low-income countries (0.1%; 3/3106). The fastest-growing characteristics over time were high-income countries (location); Europe, Asia, and North America (location); diagnosis and treatment (primary purpose); hospital/clinic and academia (lead sponsor); randomized and prospective designs; and the 1-100 and 101-1000 size categories. Only 5.6% (47/842) of completed studies had results available on ClinicalTrials.gov, and this pattern persisted. Over time, there was an increase in not only the number of newly initiated studies, but also the number of completed studies without posted results.

CONCLUSIONS

Much of the rapid growth in AI/ML studies comes from high-income countries in high-resource settings, albeit with a modest increase in upper-middle-income countries (mostly China). Lower-middle-income or low-income countries remain poorly represented. The increase in randomized or prospective designs, along with 738 large studies (n>1000), mostly ongoing, may indicate that enough studies are shifting from an in silico evaluation stage toward a prospective comparative evaluation stage. However, the ongoing limited availability of basic results on ClinicalTrials.gov contrasts with this field's rapid advancements and the public registry's role in reducing publication and outcome reporting biases.

Prospective randomized evaluation of the sustained impact of assistive artificial intelligence on anesthetists' ultrasound scanning for regional anesthesia

Objectives

Ultrasound-guided regional anesthesia (UGRA) relies on acquiring and interpreting an appropriate view of sonoanatomy. Artificial intelligence (AI) has the potential to aid this by applying a color overlay to key sonoanatomical structures.The primary aim was to determine whether an AI-generated color overlay was associated with a difference in participants' ability to identify an appropriate block view over a 2-month period after a standardized teaching session (as judged by a blinded assessor). Secondary outcomes included the ability to identify an appropriate block view (unblinded assessor), global rating score and participant confidence scores.

Design

Randomized, partially blinded, prospective cross-over study.

Setting

Simulation scans on healthy volunteers. Initial assessments on 29 November 2022 and 30 November 2022, with follow-up on 25 January 2023 - 27 January 2023.

Participants

57 junior anesthetists undertook initial assessments and 51 (89.47%) returned at 2 months.

Intervention

Participants performed ultrasound scans for six peripheral nerve blocks, with AI assistance randomized to half of the blocks. Cross-over assignment was employed for 2 months.

Main outcome measures

Blinded experts assessed whether the block view acquired was acceptable (yes/no). Unblinded experts also assessed this parameter and provided a global performance rating (0-100). Participants reported scan confidence (0-100).

Results

AI assistance was associated with a higher rate of appropriate block view acquisition in both blinded and unblinded assessments (p=0.02 and <0.01, respectively). Participant confidence and expert rating scores were superior throughout (all p<0.01).

Conclusions

Assistive AI was associated with superior ultrasound scanning performance 2 months after formal teaching. It may aid application of sonoanatomical knowledge and skills gained in teaching, to support delivery of UGRA beyond the immediate post-teaching period.

Trial registration number

www.clinicaltrials.govNCT05583032.

Predicting blood transfusion following traumatic injury using machine learning models: A systematic review and narrative synthesis

BACKGROUND

Hemorrhage is a leading cause of preventable death in trauma. Accurately predicting a patient's blood transfusion requirement is essential but can be difficult. Machine learning (ML) is a field of artificial intelligence that is emerging within medicine for accurate prediction modeling. This systematic review aimed to identify and evaluate all ML models that predict blood transfusion in trauma.

METHODS

This systematic review was registered on the International Prospective register of Systematic Reviews (CRD4202237110). MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials were systematically searched. Publications reporting an ML model that predicted blood transfusion in injured adult patients were included. Data extraction and risk of bias assessment were performed using validated frameworks. Data were synthesized narratively because of significant heterogeneity.

RESULTS

Twenty-five ML models for blood transfusion prediction in trauma were identified. Models incorporated diverse predictors and varied ML methodologies. Predictive performance was variable, but eight models achieved excellent discrimination (area under the receiver operating characteristic curve, >0.9) and nine models achieved good discrimination (area under the receiver operating characteristic curve, >0.8) in internal validation. Only two models reported measures of calibration. Four models have been externally validated in prospective cohorts: the Bleeding Risk Index, Compensatory Reserve Index, the Marsden model, and the Mina model. All studies were considered at high risk of bias often because of retrospective data sets, small sample size, and lack of external validation.

DISCUSSION

This review identified 25 ML models developed to predict blood transfusion requirement after injury. Seventeen ML models demonstrated good to excellent performance in silico, but only four models were externally validated. To date, ML models demonstrate the potential for early and individualized blood transfusion prediction, but further research is critically required to narrow the gap between ML model development and clinical application.

LEVEL OF EVIDENCE

Systematic Review Without Meta-analysis; Level IV.

Standardization, Collaboration, and Education in the Implementation of Artificial Intelligence in Otolaryngology: The Key to Scalable Impact

The study delves into the crucial role of standardization, collaboration, and education in the integration of artificial intelligence (AI) in otolaryngology. It emphasizes the necessity of large, diverse datasets for effective AI implementation in health care, particularly in otolaryngology, due to its reliance on medical imagery and diverse instruments. The text identifies current barriers, including siloed work in academia and sparse academic-industrial partnerships, while proposing solutions like forming interdisciplinary teams and aligning incentives. Moreover, it discusses the importance of standardizing AI projects through system reporting and advocates for AI education and literacy among otolaryngology practitioners.

A Responsible Framework for Applying Artificial Intelligence on Medical Images and Signals at the Point of Care: The PACS-AI Platform

The potential of artificial intelligence (AI) in medicine lies in its ability to enhance clinicians' capacity to analyse medical images, thereby improving diagnostic precision and accuracy and thus enhancing current tests. However, the integration of AI within health care is fraught with difficulties. Heterogeneity among health care system applications, reliance on proprietary closed-source software, and rising cybersecurity threats pose significant challenges. Moreover, before their deployment in clinical settings, AI models must demonstrate their effectiveness across a wide range of scenarios and must be validated by prospective studies, but doing so requires testing in an environment mirroring the clinical workflow, which is difficult to achieve without dedicated software. Finally, the use of AI techniques in health care raises significant legal and ethical issues, such as the protection of patient privacy, the prevention of bias, and the monitoring of the device's safety and effectiveness for regulatory compliance. This review describes challenges to AI integration in health care and provides guidelines on how to move forward. We describe an open-source solution that we developed that integrates AI models into the Picture Archives Communication System (PACS), called PACS-AI. This approach aims to increase the evaluation of AI models by facilitating their integration and validation with existing medical imaging databases. PACS-AI may overcome many current barriers to AI deployment and offer a pathway toward responsible, fair, and effective deployment of AI models in health care. In addition, we propose a list of criteria and guidelines that AI researchers should adopt when publishing a medical AI model to enhance standardisation and reproducibility.

Aliado - A design concept of AI for decision support in oncological liver surgery

BACKGROUND

The interest in artificial intelligence (AI) is increasing. Systematic reviews suggest that there are many machine learning algorithms in surgery, however, only a minority of the studies integrate AI applications in clinical workflows. Our objective was to design and evaluate a concept to use different kinds of AI for decision support in oncological liver surgery along the treatment path.

METHODS

In an exploratory co-creation between design experts, surgeons, and data scientists, pain points along the treatment path were identified. Potential designs for AI-assisted solutions were developed and iteratively refined. Finally, an evaluation of the design concept was performed with n = 20 surgeons to get feedback on the different functionalities and evaluate the usability with the System Usability Scale (SUS). Participating surgeons had a mean of 14.0 ± 5.0 years of experience after graduation.

RESULTS

The design concept was named "Aliado". Five different scenarios were identified where AI could support surgeons. Mean score of SUS was 68.2 ( ± 13.6 SD). The highest valued functionalities were "individualized prediction of survival, short-term mortality and morbidity", and "individualized recommendation of surgical strategy".

CONCLUSION

Aliado is a design prototype that shows how AI could be integrated into the clinical workflow. Even without a fleshed out user interface, the SUS already yielded borderline good results. Expert surgeons rated the functionalities favorably, and most of them expressed their willingness to work with a similar application in the future. Thus, Aliado can serve as a surgical vision of how an ideal AI-based assistance could look like.

Evaluation of the impact of artificial intelligence-assisted image interpretation on the diagnostic performance of clinicians in identifying pneumothoraces on plain chest X-ray: a multi-case multi-reader study

BACKGROUND

Artificial intelligence (AI)-assisted image interpretation is a fast-developing area of clinical innovation. Most research to date has focused on the performance of AI-assisted algorithms in comparison with that of radiologists rather than evaluating the algorithms' impact on the clinicians who often undertake initial image interpretation in routine clinical practice. This study assessed the impact of AI-assisted image interpretation on the diagnostic performance of frontline acute care clinicians for the detection of pneumothoraces (PTX).

METHODS

A multicentre blinded multi-case multi-reader study was conducted between October 2021 and January 2022. The online study recruited 18 clinician readers from six different clinical specialties, with differing levels of seniority, across four English hospitals. The study included 395 plain CXR images, 189 positive for PTX and 206 negative. The reference standard was the consensus opinion of two thoracic radiologists with a third acting as arbitrator. General Electric Healthcare Critical Care Suite (GEHC CCS) PTX algorithm was applied to the final dataset. Readers individually interpreted the dataset without AI assistance, recording the presence or absence of a PTX and a confidence rating. Following a 'washout' period, this process was repeated including the AI output.

RESULTS

Analysis of the performance of the algorithm for detecting or ruling out a PTX revealed an overall AUROC of 0.939. Overall reader sensitivity increased by 11.4% (95% CI 4.8, 18.0, p=0.002) from 66.8% (95% CI 57.3, 76.2) unaided to 78.1% aided (95% CI 72.2, 84.0, p=0.002), specificity 93.9% (95% CI 90.9, 97.0) without AI to 95.8% (95% CI 93.7, 97.9, p=0.247). The junior reader subgroup showed the largest improvement at 21.7% (95% CI 10.9, 32.6), increasing from 56.0% (95% CI 37.7, 74.3) to 77.7% (95% CI 65.8, 89.7, p<0.01).

CONCLUSION

The study indicates that AI-assisted image interpretation significantly enhances the diagnostic accuracy of clinicians in detecting PTX, particularly benefiting less experienced practitioners. While overall interpretation time remained unchanged, the use of AI improved diagnostic confidence and sensitivity, especially among junior clinicians. These findings underscore the potential of AI to support less skilled clinicians in acute care settings.

Added value of artificial intelligence solutions for arterial stenosis detection on head and neck CT angiography: A randomized crossover multi-reader multi-case study

PURPOSE

The purpose of this study was to investigate the added value of artificial intelligence (AI) solutions for the detection of arterial stenosis (AS) on head and neck CT angiography (CTA).

MATERIALS AND METHODS

Patients who underwent head and neck CTA examinations at two hospitals were retrospectively included. CTA examinations were randomized into group 1 (without AI-washout-with AI) and group 2 (with AI-washout-without AI), and six readers (two radiology residents, two non-neuroradiologists, and two neuroradiologists) independently interpreted each CTA examination without and with AI solutions. Additionally, reading time was recorded for each patient. Digital subtraction angiography was used as the standard of reference. The diagnostic performance for AS at lesion and patient levels with four AS thresholds (30 %, 50 %, 70 %, and 100 %) was assessed by calculating sensitivity, false-positive lesions index (FPLI), specificity, and accuracy.

RESULTS

A total of 268 patients (169 men, 63.1 %) with a median age of 65 years (first quartile, 57; third quartile, 72; age range: 28-88 years) were included. At the lesion level, AI improved the sensitivity of all readers by 5.2 % for detecting AS ≥ 30 % (P < 0.001). Concurrently, AI reduced the FPLI of all readers and specifically neuroradiologists for detecting non-occlusive AS (all P < 0.05). At the patient level, AI improved the accuracy of all readers by 4.1 % (73.9 % [1189/1608] without AI vs. 78.0 % [1254/1608] with AI) (P < 0.001). Sensitivity for AS ≥ 30 % and the specificity for AS ≥ 70 % increased for all readers with AI assistance (P = 0.01). The median reading time for all readers was reduced from 268 s without AI to 241 s with AI (P< 0.001).

CONCLUSION

AI-assisted diagnosis improves the performance of radiologists in detecting head and neck AS, and shortens reading time.

Should AI models be explainable to clinicians?

In the high-stakes realm of critical care, where daily decisions are crucial and clear communication is paramount, comprehending the rationale behind Artificial Intelligence (AI)-driven decisions appears essential. While AI has the potential to improve decision-making, its complexity can hinder comprehension and adherence to its recommendations. "Explainable AI" (XAI) aims to bridge this gap, enhancing confidence among patients and doctors. It also helps to meet regulatory transparency requirements, offers actionable insights, and promotes fairness and safety. Yet, defining explainability and standardising assessments are ongoing challenges and balancing performance and explainability can be needed, even if XAI is a growing field.

Artificial intelligence in interventional radiology: Current concepts and future trends

While artificial intelligence (AI) is already well established in diagnostic radiology, it is beginning to make its mark in interventional radiology. AI has the potential to dramatically change the daily practice of interventional radiology at several levels. In the preoperative setting, recent advances in deep learning models, particularly foundation models, enable effective management of multimodality and increased autonomy through their ability to function minimally without supervision. Multimodality is at the heart of patient-tailored management and in interventional radiology, this translates into the development of innovative models for patient selection and outcome prediction. In the perioperative setting, AI is manifesting itself in applications that assist radiologists in image analysis and real-time decision making, thereby improving the efficiency, accuracy, and safety of interventions. In synergy with advances in robotic technologies, AI is laying the groundwork for an increased autonomy. From a research perspective, the development of artificial health data, such as AI-based data augmentation, offers an innovative solution to this central issue and promises to stimulate research in this area. This review aims to provide the medical community with the most important current and future applications of AI in interventional radiology.

Illuminating the future of precision cancer surgery with fluorescence imaging and artificial intelligence convergence

Real-time and accurate guidance for tumor resection has long been anticipated by surgeons. In the past decade, the flourishing material science has made impressive progress in near-infrared fluorophores that may fulfill this purpose. Fluorescence imaging-guided surgery shows great promise for clinical application and has undergone widespread evaluations, though it still requires continuous improvements to transition this technique from bench to bedside. Concurrently, the rapid progress of artificial intelligence (AI) has revolutionized medicine, aiding in the screening, diagnosis, and treatment of human doctors. Incorporating AI helps enhance fluorescence imaging and is poised to bring major innovations to surgical guidance, thereby realizing precision cancer surgery. This review provides an overview of the principles and clinical evaluations of fluorescence-guided surgery. Furthermore, recent endeavors to synergize AI with fluorescence imaging were presented, and the benefits of this interdisciplinary convergence were discussed. Finally, several implementation strategies to overcome technical hurdles were proposed to encourage and inspire future research to expedite the clinical application of these revolutionary technologies.

A Scoping Review of Artificial Intelligence Detection of Voice Pathology: Challenges and Opportunities

OBJECTIVE

Survey the current literature on artificial intelligence (AI) applications for detecting and classifying vocal pathology using voice recordings, and identify challenges and opportunities for advancing the field forward.

DATA SOURCES

PubMed, EMBASE, CINAHL, and Scopus databases.

REVIEW METHODS

A comprehensive literature search was performed following the Preferred Reporting Items for Systematic Reviews and Meta-analyses Extension for Scoping Reviews guidelines. Peer-reviewed journal articles in the English language were included if they used an AI approach to detect or classify pathological voices using voice recordings from patients diagnosed with vocal pathologies.

RESULTS

Eighty-two studies were included in the review between the years 2000 and 2023, with an increase in publication rate from one study per year in 2012 to 10 per year in 2022. Seventy-two studies (88%) were aimed at detecting the presence of voice pathology, 24 (29%) at classifying the type of voice pathology present, and 4 (5%) at assessing pathological voice using the Grade, Roughness, Breathiness, Asthenia, and Strain scale. Thirty-six databases were used to collect and analyze speech samples. Fourteen articles (17%) did not provide information about their AI model validation methodology. Zero studies moved beyond the preclinical and offline AI model development stages. Zero studies specified following a reporting guideline for AI research.

CONCLUSION

There is rising interest in the potential of AI technology to aid the detection and classification of voice pathology. Three challenges-and areas of opportunities-for advancing this research are heterogeneity of databases, lack of clinical validation studies, and inconsistent reporting.

Artificial intelligence in retinal screening using OCT images: A review of the last decade (2013-2023)

BACKGROUND AND OBJECTIVES

Optical coherence tomography (OCT) has ushered in a transformative era in the domain of ophthalmology, offering non-invasive imaging with high resolution for ocular disease detection. OCT, which is frequently used in diagnosing fundamental ocular pathologies, such as glaucoma and age-related macular degeneration (AMD), plays an important role in the widespread adoption of this technology. Apart from glaucoma and AMD, we will also investigate pertinent pathologies, such as epiretinal membrane (ERM), macular hole (MH), macular dystrophy (MD), vitreomacular traction (VMT), diabetic maculopathy (DMP), cystoid macular edema (CME), central serous chorioretinopathy (CSC), diabetic macular edema (DME), diabetic retinopathy (DR), drusen, glaucomatous optic neuropathy (GON), neovascular AMD (nAMD), myopia macular degeneration (MMD) and choroidal neovascularization (CNV) diseases. This comprehensive review examines the role that OCT-derived images play in detecting, characterizing, and monitoring eye diseases.

METHOD

The 2020 PRISMA guideline was used to structure a systematic review of research on various eye conditions using machine learning (ML) or deep learning (DL) techniques. A thorough search across IEEE, PubMed, Web of Science, and Scopus databases yielded 1787 publications, of which 1136 remained after removing duplicates. Subsequent exclusion of conference papers, review papers, and non-open-access articles reduced the selection to 511 articles. Further scrutiny led to the exclusion of 435 more articles due to lower-quality indexing or irrelevance, resulting in 76 journal articles for the review.

RESULTS

During our investigation, we found that a major challenge for ML-based decision support is the abundance of features and the determination of their significance. In contrast, DL-based decision support is characterized by a plug-and-play nature rather than relying on a trial-and-error approach. Furthermore, we observed that pre-trained networks are practical and especially useful when working on complex images such as OCT. Consequently, pre-trained deep networks were frequently utilized for classification tasks. Currently, medical decision support aims to reduce the workload of ophthalmologists and retina specialists during routine tasks. In the future, it might be possible to create continuous learning systems that can predict ocular pathologies by identifying subtle changes in OCT images.

A future role for health applications of large language models depends on regulators enforcing safety standards

Among the rapid integration of artificial intelligence in clinical settings, large language models (LLMs), such as Generative Pre-trained Transformer-4, have emerged as multifaceted tools that have potential for health-care delivery, diagnosis, and patient care. However, deployment of LLMs raises substantial regulatory and safety concerns. Due to their high output variability, poor inherent explainability, and the risk of so-called AI hallucinations, LLM-based health-care applications that serve a medical purpose face regulatory challenges for approval as medical devices under US and EU laws, including the recently passed EU Artificial Intelligence Act. Despite unaddressed risks for patients, including misdiagnosis and unverified medical advice, such applications are available on the market. The regulatory ambiguity surrounding these tools creates an urgent need for frameworks that accommodate their unique capabilities and limitations. Alongside the development of these frameworks, existing regulations should be enforced. If regulators fear enforcing the regulations in a market dominated by supply or development by large technology companies, the consequences of layperson harm will force belated action, damaging the potentiality of LLM-based applications for layperson medical advice.

Shaping the future of healthcare in British Columbia: Establishing provincial clinical governance for responsible deployment of artificial intelligence tools

As healthcare embraces the transformative potential of Artificial Intelligence (AI), it is imperative to safeguard patient and provider safety, equity, and trust in the healthcare system. This article outlines the approach taken by the British Columbia (BC) Provincial Health Services Authority (PHSA) to establish clinical governance for the responsible deployment of AI tools in healthcare. Leveraging its province-wide mandate and expertise, PHSA establishes the infrastructure and processes to proactively and systematically intake, assess, prioritize, and evaluate AI tools. PHSA proposes a coordinated approach in AI tool deployment in collaboration with regional health authorities to prevent duplication of efforts and ensure equitable access to existing and emerging AI tools across the province of BC, incorporating principles of anti-Indigenous racism, cultural safety, and humility. The proposed governance structure underscores the identification of clinical needs, proactive ethics review, rigorous risk assessment, data validation, transparent communication, provider training, and ongoing evaluation to ensure success.

Influence of artificial intelligence on the diagnostic performance of endoscopists in the assessment of Barrett's esophagus: a tandem randomized and video trial

BACKGROUND

This study evaluated the effect of an artificial intelligence (AI)-based clinical decision support system on the performance and diagnostic confidence of endoscopists in their assessment of Barrett's esophagus (BE).

METHODS

96 standardized endoscopy videos were assessed by 22 endoscopists with varying degrees of BE experience from 12 centers. Assessment was randomized into two video sets: group A (review first without AI and second with AI) and group B (review first with AI and second without AI). Endoscopists were required to evaluate each video for the presence of Barrett's esophagus-related neoplasia (BERN) and then decide on a spot for a targeted biopsy. After the second assessment, they were allowed to change their clinical decision and confidence level.

RESULTS

AI had a stand-alone sensitivity, specificity, and accuracy of 92.2%, 68.9%, and 81.3%, respectively. Without AI, BE experts had an overall sensitivity, specificity, and accuracy of 83.3%, 58.1%, and 71.5%, respectively. With AI, BE nonexperts showed a significant improvement in sensitivity and specificity when videos were assessed a second time with AI (sensitivity 69.8% [95%CI 65.2%-74.2%] to 78.0% [95%CI 74.0%-82.0%]; specificity 67.3% [95%CI 62.5%-72.2%] to 72.7% [95%CI 68.2%-77.3%]). In addition, the diagnostic confidence of BE nonexperts improved significantly with AI.

CONCLUSION

BE nonexperts benefitted significantly from additional AI. BE experts and nonexperts remained significantly below the stand-alone performance of AI, suggesting that there may be other factors influencing endoscopists' decisions to follow or discard AI advice.

Personalized dose selection for the first Waldenström macroglobulinemia patient on the PRECISE CURATE.AI trial

The digital revolution in healthcare, amplified by the COVID-19 pandemic and artificial intelligence (AI) advances, has led to a surge in the development of digital technologies. However, integrating digital health solutions, especially AI-based ones, in rare diseases like Waldenström macroglobulinemia (WM) remains challenging due to limited data, among other factors. CURATE.AI, a clinical decision support system, offers an alternative to big data approaches by calibrating individual treatment profiles based on that individual's data alone. We present a case study from the PRECISE CURATE.AI trial with a WM patient, where, over two years, CURATE.AI provided dynamic Ibrutinib dose recommendations to clinicians (users) aimed at achieving optimal IgM levels. An 80-year-old male with newly diagnosed WM requiring treatment due to anemia was recruited to the trial for CURATE.AI-based dosing of the Bruton tyrosine kinase inhibitor Ibrutinib. The primary and secondary outcome measures were focused on scientific and logistical feasibility. Preliminary results underscore the platform's potential in enhancing user and patient engagement, in addition to clinical efficacy. Based on a two-year-long patient enrollment into the CURATE.AI-augmented treatment, this study showcases how AI-enabled tools can support the management of rare diseases, emphasizing the integration of AI to enhance personalized therapy.

Explainable Machine Learning Approach to Prediction of Prolonged Intesive Care Unit Stay in Adult Spinal Deformity Patients: Machine Learning Outperforms Logistic Regression

STUDY DESIGN

Retrospective cohort study.

OBJECTIVES

Prolonged ICU stay is a driver of higher costs and inferior outcomes in Adult Spinal Deformity (ASD) patients. Machine learning (ML) models have recently been seen as a viable method of predicting pre-operative risk but are often 'black boxes' that do not fully explain the decision-making process. This study aims to demonstrate ML can achieve similar or greater predictive power as traditional statistical methods and follows traditional clinical decision-making processes.

METHODS

Five ML models (Decision Tree, Random Forest, Support Vector Classifier, GradBoost, and a CNN) were trained on data collected from a large urban academic center to predict whether prolonged ICU stay would be required post-operatively. 535 patients who underwent posterior fusion or combined fusion for treatment of ASD were included in each model with a 70-20-10 train-test-validation split. Further analysis was performed using Shapley Additive Explanation (SHAP) values to provide insight into each model's decision-making process.

RESULTS

The model's Area Under the Receiver Operating Curve (AUROC) ranged from 0.67 to 0.83. The Random Forest model achieved the highest score. The model considered length of surgery, complications, and estimated blood loss to be the greatest predictors of prolonged ICU stay based on SHAP values.

CONCLUSIONS

We developed a ML model that was able to predict whether prolonged ICU stay was required in ASD patients. Further SHAP analysis demonstrated our model aligned with traditional clinical thinking. Thus, ML models have strong potential to assist with risk stratification and more effective and cost-efficient care.

Clinical and Genomic-Based Decision Support System to Define the Optimal Timing of Allogeneic Hematopoietic Stem-Cell Transplantation in Patients With Myelodysplastic Syndromes

PURPOSE

Allogeneic hematopoietic stem-cell transplantation (HSCT) is the only potentially curative treatment for patients with myelodysplastic syndromes (MDS). Several issues must be considered when evaluating the benefits and risks of HSCT for patients with MDS, with the timing of transplantation being a crucial question. Here, we aimed to develop and validate a decision support system to define the optimal timing of HSCT for patients with MDS on the basis of clinical and genomic information as provided by the Molecular International Prognostic Scoring System (IPSS-M).

PATIENTS AND METHODS

We studied a retrospective population of 7,118 patients, stratified into training and validation cohorts. A decision strategy was built to estimate the average survival over an 8-year time horizon (restricted mean survival time [RMST]) for each combination of clinical and genomic covariates and to determine the optimal transplantation policy by comparing different strategies.

RESULTS

Under an IPSS-M based policy, patients with either low and moderate-low risk benefited from a delayed transplantation policy, whereas in those belonging to moderately high-, high- and very high-risk categories, immediate transplantation was associated with a prolonged life expectancy (RMST). Modeling decision analysis on IPSS-M versus conventional Revised IPSS (IPSS-R) changed the transplantation policy in a significant proportion of patients (15% of patient candidate to be immediately transplanted under an IPSS-R-based policy would benefit from a delayed strategy by IPSS-M, whereas 19% of candidates to delayed transplantation by IPSS-R would benefit from immediate HSCT by IPSS-M), resulting in a significant gain-in-life expectancy under an IPSS-M-based policy (P = .001).

CONCLUSION

These results provide evidence for the clinical relevance of including genomic features into the transplantation decision making process, allowing personalizing the hazards and effectiveness of HSCT in patients with MDS.

Evaluating Artificial Intelligence in Clinical Settings-Let Us Not Reinvent the Wheel

Given the requirement to minimize the risks and maximize the benefits of technology applications in health care provision, there is an urgent need to incorporate theory-informed health IT (HIT) evaluation frameworks into existing and emerging guidelines for the evaluation of artificial intelligence (AI). Such frameworks can help developers, implementers, and strategic decision makers to build on experience and the existing empirical evidence base. We provide a pragmatic conceptual overview of selected concrete examples of how existing theory-informed HIT evaluation frameworks may be used to inform the safe development and implementation of AI in health care settings. The list is not exhaustive and is intended to illustrate applications in line with various stakeholder requirements. Existing HIT evaluation frameworks can help to inform AI-based development and implementation by supporting developers and strategic decision makers in considering relevant technology, user, and organizational dimensions. This can facilitate the design of technologies, their implementation in user and organizational settings, and the sustainability and scalability of technologies.

Predictors of Health Care Practitioners' Intention to Use AI-Enabled Clinical Decision Support Systems: Meta-Analysis Based on the Unified Theory of Acceptance and Use of Technology

BACKGROUND

Artificial intelligence-enabled clinical decision support systems (AI-CDSSs) offer potential for improving health care outcomes, but their adoption among health care practitioners remains limited.

OBJECTIVE

This meta-analysis identified predictors influencing health care practitioners' intention to use AI-CDSSs based on the Unified Theory of Acceptance and Use of Technology (UTAUT). Additional predictors were examined based on existing empirical evidence.

METHODS

The literature search using electronic databases, forward searches, conference programs, and personal correspondence yielded 7731 results, of which 17 (0.22%) studies met the inclusion criteria. Random-effects meta-analysis, relative weight analyses, and meta-analytic moderation and mediation analyses were used to examine the relationships between relevant predictor variables and the intention to use AI-CDSSs.

RESULTS

The meta-analysis results supported the application of the UTAUT to the context of the intention to use AI-CDSSs. The results showed that performance expectancy (r=0.66), effort expectancy (r=0.55), social influence (r=0.66), and facilitating conditions (r=0.66) were positively associated with the intention to use AI-CDSSs, in line with the predictions of the UTAUT. The meta-analysis further identified positive attitude (r=0.63), trust (r=0.73), anxiety (r=-0.41), perceived risk (r=-0.21), and innovativeness (r=0.54) as additional relevant predictors. Trust emerged as the most influential predictor overall. The results of the moderation analyses show that the relationship between social influence and use intention becomes weaker with increasing age. In addition, the relationship between effort expectancy and use intention was stronger for diagnostic AI-CDSSs than for devices that combined diagnostic and treatment recommendations. Finally, the relationship between facilitating conditions and use intention was mediated through performance and effort expectancy.

CONCLUSIONS

This meta-analysis contributes to the understanding of the predictors of intention to use AI-CDSSs based on an extended UTAUT model. More research is needed to substantiate the identified relationships and explain the observed variations in effect sizes by identifying relevant moderating factors. The research findings bear important implications for the design and implementation of training programs for health care practitioners to ease the adoption of AI-CDSSs into their practice.

Artificial intelligence in liver cancer - new tools for research and patient management

Liver cancer has high incidence and mortality globally. Artificial intelligence (AI) has advanced rapidly, influencing cancer care. AI systems are already approved for clinical use in some tumour types (for example, colorectal cancer screening). Crucially, research demonstrates that AI can analyse histopathology, radiology and natural language in liver cancer, and can replace manual tasks and access hidden information in routinely available clinical data. However, for liver cancer, few of these applications have translated into large-scale clinical trials or clinically approved products. Here, we advocate for the incorporation of AI in all stages of liver cancer management. We present a taxonomy of AI approaches in liver cancer, highlighting areas with academic and commercial potential, and outline a policy for AI-based liver cancer management, including interdisciplinary training of researchers, clinicians and patients. The potential of AI in liver cancer is immense, but effort is required to ensure that AI can fulfil expectations.

A joint ESTRO and AAPM guideline for development, clinical validation and reporting of artificial intelligence models in radiation therapy

BACKGROUND AND PURPOSE

Artificial Intelligence (AI) models in radiation therapy are being developed with increasing pace. Despite this, the radiation therapy community has not widely adopted these models in clinical practice. A cohesive guideline on how to develop, report and clinically validate AI algorithms might help bridge this gap.

METHODS AND MATERIALS

A Delphi process with all co-authors was followed to determine which topics should be addressed in this comprehensive guideline. Separate sections of the guideline, including Statements, were written by subgroups of the authors and discussed with the whole group at several meetings. Statements were formulated and scored as highly recommended or recommended.

RESULTS

The following topics were found most relevant: Decision making, image analysis, volume segmentation, treatment planning, patient specific quality assurance of treatment delivery, adaptive treatment, outcome prediction, training, validation and testing of AI model parameters, model availability for others to verify, model quality assurance/updates and upgrades, ethics. Key references were given together with an outlook on current hurdles and possibilities to overcome these. 19 Statements were formulated.

CONCLUSION

A cohesive guideline has been written which addresses main topics regarding AI in radiation therapy. It will help to guide development, as well as transparent and consistent reporting and validation of new AI tools and facilitate adoption.

Clinical Evaluation of Artificial Intelligence-Enabled Interventions

Artificial intelligence (AI) health technologies are increasingly available for use in real-world care. This emerging opportunity is accompanied by a need for decision makers and practitioners across healthcare systems to evaluate the safety and effectiveness of these interventions against the needs of their own setting. To meet this need, high-quality evidence regarding AI-enabled interventions must be made available, and decision makers in varying roles and settings must be empowered to evaluate that evidence within the context in which they work. This article summarizes good practices across four stages of evidence generation for AI health technologies: study design, study conduct, study reporting, and study appraisal.

The TRIPOD-LLM Statement: A Targeted Guideline For Reporting Large Language Models Use

Large Language Models (LLMs) are rapidly being adopted in healthcare, necessitating standardized reporting guidelines. We present TRIPOD-LLM, an extension of the TRIPOD+AI statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion. The guidelines introduce a modular format accommodating various LLM research designs and tasks, with 14 main items and 32 subitems applicable across all categories. Developed through an expedited Delphi process and expert consensus, TRIPOD-LLM emphasizes transparency, human oversight, and task-specific performance reporting. We also introduce an interactive website ( https://tripod-llm.vercel.app/ ) facilitating easy guideline completion and PDF generation for submission. As a living document, TRIPOD-LLM will evolve with the field, aiming to enhance the quality, reproducibility, and clinical applicability of LLM research in healthcare through comprehensive reporting.

COI

DSB: Editorial, unrelated to this work: Associate Editor of Radiation Oncology, HemOnc.org (no financial compensation); Research funding, unrelated to this work: American Association for Cancer Research; Advisory and consulting, unrelated to this work: MercurialAI. DDF: Editorial, unrelated to this work: Associate Editor of JAMIA, Editorial Board of Scientific Data, Nature; Funding, unrelated to this work: the intramural research program at the U.S. National Library of Medicine, National Institutes of Health. JWG: Editorial, unrelated to this work: Editorial Board of Radiology: Artificial Intelligence, British Journal of Radiology AI journal and NEJM AI. All other authors declare no conflicts of interest.

Adherence of studies involving artificial intelligence in the analysis of ophthalmology electronic medical records to AI-specific items from the CONSORT-AI guideline: a systematic review

PURPOSE

In the context of ophthalmologic practice, there has been a rapid increase in the amount of data collected using electronic health records (EHR). Artificial intelligence (AI) offers a promising means of centralizing data collection and analysis, but to date, most AI algorithms have only been applied to analyzing image data in ophthalmologic practice. In this review we aimed to characterize the use of AI in the analysis of EHR, and to critically appraise the adherence of each included study to the CONSORT-AI reporting guideline.

METHODS

A comprehensive search of three relevant databases (MEDLINE, EMBASE, and Cochrane Library) from January 2010 to February 2023 was conducted. The included studies were evaluated for reporting quality based on the AI-specific items from the CONSORT-AI reporting guideline.

RESULTS

Of the 4,968 articles identified by our search, 89 studies met all inclusion criteria and were included in this review. Most of the studies utilized AI for ocular disease prediction (n = 41, 46.1%), and diabetic retinopathy was the most studied ocular pathology (n = 19, 21.3%). The overall mean CONSORT-AI score across the 14 measured items was 12.1 (range 8-14, median 12). Categories with the lowest adherence rates were: describing handling of poor quality data (48.3%), specifying participant inclusion and exclusion criteria (56.2%), and detailing access to the AI intervention or its code, including any restrictions (62.9%).

CONCLUSIONS

In conclusion, we have identified that AI is prominently being used for disease prediction in ophthalmology clinics, however these algorithms are limited by their lack of generalizability and cross-center reproducibility. A standardized framework for AI reporting should be developed, to improve AI applications in the management of ocular disease and ophthalmology decision making.

Accuracy and safety of an autonomous artificial intelligence clinical assistant conducting telemedicine follow-up assessment for cataract surgery

Background

Artificial intelligence deployed to triage patients post-cataract surgery could help to identify and prioritise individuals who need clinical input and to expand clinical capacity. This study investigated the accuracy and safety of an autonomous telemedicine call (Dora, version R1) in detecting cataract surgery patients who need further management and compared its performance against ophthalmic specialists.

Methods

225 participants were recruited from two UK public teaching hospitals after routine cataract surgery between 17 September 2021 and 31 January 2022. Eligible patients received a call from Dora R1 to conduct a follow-up assessment approximately 3 weeks post cataract surgery, which was supervised in real-time by an ophthalmologist. The primary analysis compared decisions made independently by Dora R1 and the supervising ophthalmologist about the clinical significance of five symptoms and whether the patient required further review. Secondary analyses used mixed methods to examine Dora R1's usability and acceptability and to assess cost impact compared to standard care. This study is registered with ClinicalTrials.gov (NCT05213390) and ISRCTN (16038063).

Findings

202 patients were included in the analysis, with data collection completed on 23 March 2022. Dora R1 demonstrated an overall outcome sensitivity of 94% and specificity of 86% and showed moderate to strong agreement (kappa: 0.758-0.970) with clinicians in all parameters. Safety was validated by assessing subsequent outcomes: 11 of the 117 patients (9%) recommended for discharge by Dora R1 had unexpected management changes, but all were also recommended for discharge by the supervising clinician. Four patients were recommended for discharge by Dora R1 but not the clinician; none required further review on callback. Acceptability, from interviews with 20 participants, was generally good in routine circumstances but patients were concerned about the lack of a 'human element' in cases with complications. Feasibility was demonstrated by the high proportion of calls completed autonomously (195/202, 96.5%). Staff cost benefits for Dora R1 compared to standard care were £35.18 per patient.

Interpretation

The composite of mixed methods analysis provides preliminary evidence for the safety, acceptability, feasibility, and cost benefits for clinical adoption of an artificial intelligence conversational agent, Dora R1, to conduct follow-up assessment post-cataract surgery. Further evaluation in real-world implementation should be conducted to provide additional evidence around safety and effectiveness in a larger sample from a more diverse set of Trusts.

Funding

This manuscript is independent research funded by the National Institute for Health Research and NHSX (Artificial Intelligence in Health and Care Award, AI_AWARD01852).

Improving the efficiency and accuracy of cardiovascular magnetic resonance with artificial intelligence-review of evidence and proposition of a roadmap to clinical translation

BACKGROUND

Cardiovascular magnetic resonance (CMR) is an important imaging modality for the assessment of heart disease; however, limitations of CMR include long exam times and high complexity compared to other cardiac imaging modalities. Recently advancements in artificial intelligence (AI) technology have shown great potential to address many CMR limitations. While the developments are remarkable, translation of AI-based methods into real-world CMR clinical practice remains at a nascent stage and much work lies ahead to realize the full potential of AI for CMR.

METHODS

Herein we review recent cutting-edge and representative examples demonstrating how AI can advance CMR in areas such as exam planning, accelerated image reconstruction, post-processing, quality control, classification and diagnosis.

RESULTS

These advances can be applied to speed up and simplify essentially every application including cine, strain, late gadolinium enhancement, parametric mapping, 3D whole heart, flow, perfusion and others. AI is a unique technology based on training models using data. Beyond reviewing the literature, this paper discusses important AI-specific issues in the context of CMR, including (1) properties and characteristics of datasets for training and validation, (2) previously published guidelines for reporting CMR AI research, (3) considerations around clinical deployment, (4) responsibilities of clinicians and the need for multi-disciplinary teams in the development and deployment of AI in CMR, (5) industry considerations, and (6) regulatory perspectives.

CONCLUSIONS

Understanding and consideration of all these factors will contribute to the effective and ethical deployment of AI to improve clinical CMR.

Scoping review of clinical decision support systems for multiple sclerosis management: Leveraging information technology and massive health data

BACKGROUND AND PURPOSE

Multiple sclerosis (MS) is a complex autoimmune disease of the central nervous system, with numerous therapeutic options, but a lack of biomarkers to support a mechanistic approach to precision medicine. A computational approach to precision medicine could proceed from clinical decision support systems (CDSSs). They are digital tools aiming to empower physicians through the clinical applications of information technology and massive data. However, the process of their clinical development is still maturing; we aimed to review it in the field of MS.

METHODS

For this scoping review, we screened systematically the PubMed database. We identified 24 articles reporting 14 CDSS projects and compared their technical and software development aspects.

RESULTS

The projects position themselves in various contexts of usage with various algorithmic approaches: expert systems, CDSSs based on similar patients' data visualization, and model-based CDSSs implementing mathematical predictive models. So far, no project has completed its clinical development up to certification for clinical use with global release. Some CDSSs have been replaced at subsequent project iterations. The most advanced projects did not necessarily report every step of clinical development in a dedicated article (proof of concept, offline validation, refined prototype, live clinical evaluation, comparative prospective evaluation). They seek different software distribution options to integrate into health care: internal usage, "peer-to-peer," and marketing distribution.

CONCLUSIONS

This review illustrates the potential of clinical applications of information technology and massive data to support MS management and helps clarify the roadmap for future projects as a multidisciplinary and multistep process.

Detection and Grading of Radiographic Hand Osteoarthritis Using an Automated Machine Learning Platform

OBJECTIVE

Automated machine learning (autoML) platforms allow health care professionals to play an active role in the development of machine learning (ML) algorithms according to scientific or clinical needs. The aim of this study was to develop and evaluate such a model for automated detection and grading of distal hand osteoarthritis (OA).

METHODS

A total of 13,690 hand radiographs from 2,863 patients within the Swiss Cohort of Quality Management (SCQM) and an external control data set of 346 non-SCQM patients were collected and scored for distal interphalangeal OA (DIP-OA) using the modified Kellgren/Lawrence (K/L) score. Giotto (Learn to Forecast [L2F]) was used as an autoML platform for training two convolutional neural networks for DIP joint extraction and subsequent classification according to the K/L scores. A total of 48,892 DIP joints were extracted and then used to train the classification model. Heatmaps were generated independently of the platform. User experience of a web application as a provisional user interface was investigated by rheumatologists and radiologists.

RESULTS

The sensitivity and specificity of this model for detecting DIP-OA were 79% and 86%, respectively. The accuracy for grading the correct K/L score was 75%, with a κ score of 0.76. The accuracy per DIP-OA class differed, with 86% for no OA (defined as K/L scores 0 and 1), 71% for a K/L score of 2, 46% for a K/L score of 3, and 67% for a K/L score of 4. Similar values were obtained in an independent external test set. Qualitative and quantitative user experience testing of the web application revealed a moderate to high demand for automated DIP-OA scoring among rheumatologists. Conversely, radiologists expressed a low demand, except for the use of heatmaps.

CONCLUSION

AutoML platforms are an opportunity to develop clinical end-to-end ML algorithms. Here, automated radiographic DIP-OA detection is both feasible and usable, whereas grading among individual K/L scores (eg, for clinical trials) remains challenging.

Large language models for science and medicine

Large language models (LLMs) are a type of machine learning model that learn statistical patterns over text, such as predicting the next words in a sequence of text. Both general purpose and task-specific LLMs have demonstrated potential across diverse applications. Science and medicine have many data types that are highly suitable for LLMs, such as scientific texts (publications, patents and textbooks), electronic medical records, large databases of DNA and protein sequences and chemical compounds. Carefully validated systems that can understand and reason across all these modalities may maximize benefits. Despite the inevitable limitations and caveats of any new technology and some uncertainties specific to LLMs, LLMs have the potential to be transformative in science and medicine.

[Digitalization and clinical decision tools]

Digitalization in cardiovascular emergencies is rapidly evolving, analogous to the development in medicine, driven by the increasingly broader availability of digital structures and improved networks, electronic health records and the interconnectivity of systems. The potential use of digital health in patients with acute chest pain starts even in the prehospital phase with the transmission of a digital electrocardiogram (ECG) as well as telemedical support and digital emergency management, which facilitate optimization of the rescue pathways and reduce critical time intervals. The increasing dissemination and acceptance of guideline apps and clinical decision support tools as well as integrated calculators and electronic scores are anticipated to improve guideline adherence, translating into a better quality of treatment and improved outcomes. Implementation of artificial intelligence to support image analysis and also the prediction of coronary artery stenosis requiring interventional treatment or impending cardiovascular events, such as heart attacks or death, have an enormous potential especially as conventional instruments frequently yield suboptimal results; however, there are barriers to the rapid dissemination of corresponding decision aids, such as the regulatory rules related to approval as a medical product, data protection issues and other legal liability aspects, which must be considered.

Protocol for the development of the Chatbot Assessment Reporting Tool (CHART) for clinical advice

INTRODUCTION

Large language model (LLM)-linked chatbots are being increasingly applied in healthcare due to their impressive functionality and public availability. Studies have assessed the ability of LLM-linked chatbots to provide accurate clinical advice. However, the methods applied in these Chatbot Assessment Studies are inconsistent due to the lack of reporting standards available, which obscures the interpretation of their study findings. This protocol outlines the development of the Chatbot Assessment Reporting Tool (CHART) reporting guideline.

METHODS AND ANALYSIS

The development of the CHART reporting guideline will consist of three phases, led by the Steering Committee. During phase one, the team will identify relevant reporting guidelines with artificial intelligence extensions that are published or in development by searching preprint servers, protocol databases, and the Enhancing the Quality and Transparency of health research Network. During phase two, we will conduct a scoping review to identify studies that have addressed the performance of LLM-linked chatbots in summarising evidence and providing clinical advice. The Steering Committee will identify methodology used in previous Chatbot Assessment Studies. Finally, the study team will use checklist items from prior reporting guidelines and findings from the scoping review to develop a draft reporting checklist. We will then perform a Delphi consensus and host two synchronous consensus meetings with an international, multidisciplinary group of stakeholders to refine reporting checklist items and develop a flow diagram.

ETHICS AND DISSEMINATION

We will publish the final CHART reporting guideline in peer-reviewed journals and will present findings at peer-reviewed meetings. Ethical approval was submitted to the Hamilton Integrated Research Ethics Board and deemed "not required" in accordance with the Tri-Council Policy Statement (TCPS2) for the development of the CHART reporting guideline (#17025).

REGISTRATION

This study protocol is preregistered with Open Science Framework: https://doi.org/10.17605/OSF.IO/59E2Q.

Leading in the development, standardised evaluation, and adoption of artificial intelligence in clinical practice: regional anaesthesia as an example

A recent study by Suissa and colleagues explored the clinical relevance of a medical image segmentation metric (Dice metric) commonly used in the field of artificial intelligence (AI). They showed that pixel-wise agreement for physician identification of structures on ultrasound images is variable, and a relatively low Dice metric (0.34) correlated to a substantial agreement on subjective clinical assessment. We highlight the need to bring structure and clinical perspective to the evaluation of medical AI, which clinicians are best placed to direct.

Realizing the promise of Project Optimus: Challenges and emerging opportunities for dose optimization in oncology drug development

Project Optimus is a US Food and Drug Administration Oncology Center of Excellence initiative aimed at reforming the dose selection and optimization paradigm in oncology drug development. This project seeks to bring together pharmaceutical companies, international regulatory agencies, academic institutions, patient advocates, and other stakeholders. Although there is much promise in this initiative, there are several challenges that need to be addressed, including multidimensionality of the dose optimization problem in oncology, the heterogeneity of cancer and patients, importance of evaluating long-term tolerability beyond dose-limiting toxicities, and the lack of reliable biomarkers for long-term efficacy. Through the lens of Totality of Evidence and with the mindset of model-informed drug development, we offer insights into dose optimization by building a quantitative knowledge base integrating diverse sources of data and leveraging quantitative modeling tools to build evidence for drug dosage considering exposure, disease biology, efficacy, toxicity, and patient factors. We believe that rational dose optimization can be achieved in oncology drug development, improving patient outcomes by maximizing therapeutic benefit while minimizing toxicity.

Can Machine Learning Personalize Cardiovascular Therapy in Sepsis?

Large randomized trials in sepsis have generally failed to find effective novel treatments. This is increasingly attributed to patient heterogeneity, including heterogeneous cardiovascular changes in septic shock. We discuss the potential for machine learning systems to personalize cardiovascular resuscitation in sepsis. While the literature is replete with proofs of concept, the technological readiness of current systems is low, with a paucity of clinical trials and proven patient benefit. Systems may be vulnerable to confounding and poor generalization to new patient populations or contemporary patterns of care. Typical electronic health records do not capture rich enough data, at sufficient temporal resolution, to produce systems that make actionable treatment suggestions. To resolve these issues, we recommend a simultaneous focus on technical challenges and removing barriers to translation. This will involve improving data quality, adopting causally grounded models, prioritizing safety assessment and integration into healthcare workflows, conducting randomized clinical trials and aligning with regulatory requirements.

Artificial intelligence for ultrasound scanning in regional anaesthesia: a scoping review of the evidence from multiple disciplines

BACKGROUND

Artificial intelligence (AI) for ultrasound scanning in regional anaesthesia is a rapidly developing interdisciplinary field. There is a risk that work could be undertaken in parallel by different elements of the community but with a lack of knowledge transfer between disciplines, leading to repetition and diverging methodologies. This scoping review aimed to identify and map the available literature on the accuracy and utility of AI systems for ultrasound scanning in regional anaesthesia.

METHODS

A literature search was conducted using Medline, Embase, CINAHL, IEEE Xplore, and ACM Digital Library. Clinical trial registries, a registry of doctoral theses, regulatory authority databases, and websites of learned societies in the field were searched. Online commercial sources were also reviewed.

RESULTS

In total, 13,014 sources were identified; 116 were included for full-text review. A marked change in AI techniques was noted in 2016-17, from which point on the predominant technique used was deep learning. Methods of evaluating accuracy are variable, meaning it is impossible to compare the performance of one model with another. Evaluations of utility are more comparable, but predominantly gained from the simulation setting with limited clinical data on efficacy or safety. Study methodology and reporting lack standardisation.

CONCLUSIONS

There is a lack of structure to the evaluation of accuracy and utility of AI for ultrasound scanning in regional anaesthesia, which hinders rigorous appraisal and clinical uptake. A framework for consistent evaluation is needed to inform model evaluation, allow comparison between approaches/models, and facilitate appropriate clinical adoption.

Current perspectives on the use of artificial intelligence in critical patient safety

Intensive Care Units (ICUs) have undergone enhancements in patient safety, and artificial intelligence (AI) emerges as a disruptive technology offering novel opportunities. While the published evidence is limited and presents methodological issues, certain areas show promise, such as decision support systems, detection of adverse events, and prescription error identification. The application of AI in safety may pursue predictive or diagnostic objectives. Implementing AI-based systems necessitates procedures to ensure secure assistance, addressing challenges including trust in such systems, biases, data quality, scalability, and ethical and confidentiality considerations. The development and application of AI demand thorough testing, encompassing retrospective data assessments, real-time validation with prospective cohorts, and efficacy demonstration in clinical trials. Algorithmic transparency and explainability are essential, with active involvement of clinical professionals being crucial in the implementation process.

Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis

BACKGROUND

The field of Artificial Intelligence (AI) holds transformative potential in medicine. However, the lack of universal reporting guidelines poses challenges in ensuring the validity and reproducibility of published research studies in this field.

METHODS

Based on a systematic review of academic publications and reporting standards demanded by both international consortia and regulatory stakeholders as well as leading journals in the fields of medicine and medical informatics, 26 reporting guidelines published between 2009 and 2023 were included in this analysis. Guidelines were stratified by breadth (general or specific to medical fields), underlying consensus quality, and target research phase (preclinical, translational, clinical) and subsequently analyzed regarding the overlap and variations in guideline items.

RESULTS

AI reporting guidelines for medical research vary with respect to the quality of the underlying consensus process, breadth, and target research phase. Some guideline items such as reporting of study design and model performance recur across guidelines, whereas other items are specific to particular fields and research stages.

CONCLUSIONS

Our analysis highlights the importance of reporting guidelines in clinical AI research and underscores the need for common standards that address the identified variations and gaps in current guidelines. Overall, this comprehensive overview could help researchers and public stakeholders reinforce quality standards for increased reliability, reproducibility, clinical validity, and public trust in AI research in healthcare. This could facilitate the safe, effective, and ethical translation of AI methods into clinical applications that will ultimately improve patient outcomes.

Use of artificial intelligence in critical care: opportunities and obstacles

BACKGROUND

Perhaps nowhere else in the healthcare system than in the intensive care unit environment are the challenges to create useful models with direct time-critical clinical applications more relevant and the obstacles to achieving those goals more massive. Machine learning-based artificial intelligence (AI) techniques to define states and predict future events are commonplace activities of modern life. However, their penetration into acute care medicine has been slow, stuttering and uneven. Major obstacles to widespread effective application of AI approaches to the real-time care of the critically ill patient exist and need to be addressed.

MAIN BODY

Clinical decision support systems (CDSSs) in acute and critical care environments support clinicians, not replace them at the bedside. As will be discussed in this review, the reasons are many and include the immaturity of AI-based systems to have situational awareness, the fundamental bias in many large databases that do not reflect the target population of patient being treated making fairness an important issue to address and technical barriers to the timely access to valid data and its display in a fashion useful for clinical workflow. The inherent "black-box" nature of many predictive algorithms and CDSS makes trustworthiness and acceptance by the medical community difficult. Logistically, collating and curating in real-time multidimensional data streams of various sources needed to inform the algorithms and ultimately display relevant clinical decisions support format that adapt to individual patient responses and signatures represent the efferent limb of these systems and is often ignored during initial validation efforts. Similarly, legal and commercial barriers to the access to many existing clinical databases limit studies to address fairness and generalizability of predictive models and management tools.

CONCLUSIONS

AI-based CDSS are evolving and are here to stay. It is our obligation to be good shepherds of their use and further development.

Evaluating the Reliability of a Remote Acuity Prediction Tool in a Canadian Academic Emergency Department

STUDY OBJECTIVE

There is increasing interest in harnessing artificial intelligence to virtually triage patients seeking care. The objective was to examine the reliability of a virtual machine learning algorithm to remotely predict acuity scores for patients seeking emergency department (ED) care by applying the algorithm to retrospective ED data.

METHODS

This was a retrospective review of adult patients conducted at an academic tertiary care ED (annual census 65,000) from January 2021 to August 2022. Data including ED visit date and time, patient age, sex, reason for visit, presenting complaint and patient-reported pain score were used by the machine learning algorithm to predict acuity scores. The algorithm was designed to up-triage high-risk complaints to promote safety for remote use. The predicted scores were then compared to nurse-led triage scores previously derived in real time using the electronic Canadian Triage and Acuity Scale (eCTAS), an electronic triage decision-support tool used in the ED. Interrater reliability was estimated using kappa statistics with 95% confidence intervals (CIs).

RESULTS

In total, 21,469 unique ED patient encounters were included. Exact modal agreement was achieved for 10,396 (48.4%) patient encounters. Interrater reliability ranged from poor to fair, as estimated using unweighted kappa (0.18, 95% CI 0.17 to 0.19), linear-weighted kappa (0.25, 95% CI 0.24 to 0.26), and quadratic-weighted kappa (0.36, 95% CI 0.35 to 0.37) statistics. Using the nurse-led eCTAS score as the reference, the machine learning algorithm overtriaged 9,897 (46.1%) and undertriaged 1,176 (5.5%) cases. Some of the presenting complaints under-triaged were conditions generally requiring further probing to delineate their nature, including abnormal lab/imaging results, visual disturbance, and fever.

CONCLUSION

This machine learning algorithm needs further refinement before being safely implemented for patient use.