Impact of a machine learning based decision support for Urinary Tract Infections: Prospective observational study in 36 primary care practices (Preprint)

Artificial intelligence and prescription of antibiotic therapy: present and future

INTRODUCTION

In the past few years, the use of artificial intelligence in healthcare has grown exponentially. Prescription of antibiotics is not exempt from its rapid diffusion, and various machine learning (ML) techniques, from logistic regression to deep neural networks and large language models, have been explored in the literature to support decisions regarding antibiotic prescription.

AREAS COVERED

In this narrative review, we discuss promises and challenges of the application of ML-based clinical decision support systems (ML-CDSSs) for antibiotic prescription. A search was conducted in PubMed up to April 2024.

EXPERT OPINION

Prescribing antibiotics is a complex process involving various dynamic phases. In each of these phases, the support of ML-CDSSs has shown the potential, and also the actual ability in some studies, to favorably impacting relevant clinical outcomes. Nonetheless, before widely exploiting this massive potential, there are still crucial challenges ahead that are being intensively investigated, pertaining to the transparency of training data, the definition of the sufficient degree of prediction explanations when predictions are obtained through black box models, and the legal and ethical framework for decision responsibility whenever an antibiotic prescription is supported by ML-CDSSs.

Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care: A Systematic Review

Importance

The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.

Objectives

To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.

Evidence Review

PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.

Findings

The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).

Conclusions and Relevance

The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms' quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.

Artificial-Intelligence-Based Clinical Decision Support Systems in Primary Care: A Scoping Review of Current Clinical Implementations

Primary Care Physicians (PCPs) are the first point of contact in healthcare. Because PCPs face the challenge of managing diverse patient populations while maintaining up-to-date medical knowledge and updated health records, this study explores the current outcomes and effectiveness of implementing Artificial Intelligence-based Clinical Decision Support Systems (AI-CDSSs) in Primary Healthcare (PHC). Following the PRISMA-ScR guidelines, we systematically searched five databases, PubMed, Scopus, CINAHL, IEEE, and Google Scholar, and manually searched related articles. Only CDSSs powered by AI targeted to physicians and tested in real clinical PHC settings were included. From a total of 421 articles, 6 met our criteria. We found AI-CDSSs from the US, Netherlands, Spain, and China whose primary tasks included diagnosis support, management and treatment recommendations, and complication prediction. Secondary objectives included lessening physician work burden and reducing healthcare costs. While promising, the outcomes were hindered by physicians' perceptions and cultural settings. This study underscores the potential of AI-CDSSs in improving clinical management, patient satisfaction, and safety while reducing physician workload. However, further work is needed to explore the broad spectrum of applications that the new AI-CDSSs have in several PHC real clinical settings and measure their clinical outcomes.

Smart Diagnosis of Urinary Tract Infections: is Artificial Intelligence the Fast-Lane Solution?

PURPOSE OF REVIEW

Artificial intelligence (AI) can significantly improve physicians' workflow when examining patients with UTI. However, most contemporary reviews are focused on examining the usage of AI with a restricted quantity of data, analyzing only a subset of AI algorithms, or performing narrative work without analyzing all dedicated studies. Given the preceding, the goal of this work was to conduct a mini-review to determine the current state of AI-based systems as a support in UTI diagnosis.

RECENT FINDINGS

There are sufficient publications to comprehend the potential applications of artificial intelligence in the diagnosis of UTIs. Existing research in this field, in general, publishes performance metrics that are exemplary. However, upon closer inspection, many of the available publications are burdened with flaws associated with the improper use of artificial intelligence, such as the use of a small number of samples, their lack of heterogeneity, and the absence of external validation. AI-based models cannot be classified as full-fledged physician assistants in diagnosing UTIs due to the fact that these limitations and flaws represent only a portion of all potential obstacles. Instead, such studies should be evaluated as exploratory, with a focus on the importance of future work that complies with all rules governing the use of AI. AI algorithms have demonstrated their potential for UTI diagnosis. However, further studies utilizing large, heterogeneous, prospectively collected datasets, as well as external validations, are required to define the actual clinical workflow value of artificial intelligence.

Comments by Microbiologists for Interpreting Antimicrobial Susceptibility Testing and Improving the Appropriateness of Antibiotic Therapy in Community-Acquired Urinary Tract Infections: A Randomized Double-Blind Digital Case-Vignette Controlled Superiority Trial

In primary care, urinary tract infections (UTIs) account for the majority of antibiotic prescriptions. Comments from microbiologists on interpreting the antimicrobial susceptibility testing (AST) profile for urinalysis were made to improve the prescription of antibiotics. We aimed to explore the added value of these comments on the quality of antibiotic prescribing by a superior double-blind digital randomized case-vignette trial among French general practitioners (GPs). One case vignette with (intervention) or without (control) a 'comment' after AST was randomly assigned to GPs. Among 815 participating GPs, 64.7% were women, at an average age of 37 years. Most (90.1%) used a computerized decision support system for prescribing antibiotics. Empirical antibiotic therapy was appropriate in 71.9% (95% CI, 68.8-75.0) of the cases, without differences between arms. The overall appropriateness of targeted antibiotic therapy (primary outcome) was not significantly increased when providing 'comments': 83.4% vs. 79.9% (OR = 1.26, 95% CI, 0.86-1.85). With the multivariate analysis, the appropriateness was improved by 2-folds (OR = 2.38, 95% CI, 1.02-6.16) among physicians working in healthcare facilities. Among digital-affine young general practitioners, the adjunction of a 'comment' by a microbiologist to interpret urinalysis in community-acquired UTIs did not improve the overall level of appropriateness of the targeted antibiotic.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.