An overview of clinical decision support systems: benefits, risks, and strategies for success

Do clinical decision support tools improve quality of care outcomes in the primary prevention of cardiovascular disease: A systematic review and meta-analysis

Aim

To assess the effectiveness of Clinical Decision Support Tools (CDSTs) in enhancing the quality of care outcomes in primary cardiovascular disease (CVD) prevention.

Methods

A systematic review was undertaken in accordance with PRISMA guidelines, and included searches in Ovid Medline, Ovid Embase, CINAHL, and Scopus. Eligible studies were randomized controlled trials of CDSTs comprising digital notifications in electronic health systems (EHS/EHR) in various primary healthcare settings, published post-2013, in patients with CVD risks and without established CVD. Two reviewers independently assessed risk of bias using the Cochrane RoB-2 tool. Attainment of clinical targets was analysed using a Restricted Maximum Likelihood random effects meta-analysis. Other relevant outcomes were narratively synthesised due to heterogeneity of studies and outcome metrics.

Results

Meta-analysis revealed CDSTs showed improvement in systolic (Mean Standardised Difference (MSD)=0.39, 95 %CI=-0.31, -1.10) and diastolic blood pressure target achievement (MSD=0.34, 95 %CI=-0.24, -0.92), but had no significant impact on lipid (MSD=0.01; 95 %CI=-0.10, 0.11) or glucose target attainment (MSD=-0.19, 95 %CI=-0.66, 0.28). The CDSTs with active prompts increased statin initiation and improved patients' adherence to clinical appointments but had minimal effect on other medications and on enhancing adherence to medication.

Conclusion

CDSTs were found to be effective in improving blood pressure clinical target attainments. However, the presence of multi-layered barriers affecting the uptake, longer-term use and active engagement from both clinicians and patients may hinder the full potential for achieving other quality of care outcomes.

Lay Summary

The study aimed to evaluate how Clinical Decision Support Tools (CDSTs) impact the quality of care for primary cardiovascular disease (CVD) management. CDSTs are tools designed to support healthcare professionals in delivering the best possible care to patients by providing timely and relevant information at the point of care (ie. digital notifications in electronic health systems). Although CDST are designed to improve the quality of healthcare outcomes, the current evidence of their effectiveness is inconsistent. Therefore, we conducted a systematic review with meta-analysis, to quantify the effectiveness of CDSTs. The eligibility criteria targeted patients with CVD risk factors, but without diagnosed CVD. The meta-analysis found that CDSTs showed improvement in systolic and diastolic blood pressure target achievement but did not significantly impact lipid or glucose target attainment. Specifically, CDSTs showed effectiveness in increasing statin prescribing but not antihypertensives or antidiabetics prescribing. Interventions with CDSTs aimed at increasing screening programmes were effective for patients with kidney diseases and high-risk patients, but not for patients with diabetes or teenage patients with hypertension. Alerts were effective in improving patients' adherence to clinical appointments but not in medication adherence. This study suggests CDSTs are effective in enhancing a limited number of quality of care outcomes in primary CVD prevention, but there is need for future research to explore the mechanisms and context of multiple barriers that may hinder the full potential for cardiovascular health outcomes to be achieved.

Hospital antimicrobial stewardship team perceptions and usability of a computerized clinical decision support system

BACKGROUND

Antimicrobial stewardship (AMS) programs aim to optimize antibiotic use through a panel of interventions. The implementation of computerized clinical decision support systems (CDSSs) offers new opportunities for semiautomated antimicrobial review by AMS teams. This study aimed to evaluate the perceived facilitators, barriers and benefits of end-users related to a commercial CDSS recently implemented in a hospital and to assess its usability.

METHODS

A mixed-method approach was used among AMS team members nine months after the implementation of the CDSS in a university hospital in northeastern France. A qualitative analysis based on individual semistructured interviews was conducted to collect end-users' perceptions. A quantitative analysis was performed using the System Usability Scale (SUS).

RESULTS

Eleven AMS team members agreed to participate. The qualitative analysis revealed technical, organizational and human barriers and facilitators of CDSS implementation. Effective collaboration with information technology teams was crucial for ensuring the installation and configuration of the software. CDSS adoption by the AMS team required time, human resources, training, adaptation and a clinical leader. Moreover, the CDSS had to be well designed, user-friendly and provide benefits to AMS activities. The quantitative analysis indicated that the CDSS was a "good" system in terms of perceived ease of use (median SUS score: 77.5/100).

CONCLUSIONS

This study shows the value of the studied CDSS to support AMS activities. It reveals barriers, facilitators and benefits to the implementation and adoption of the CDSS. These barriers and facilitators could be considered to facilitate the implementation of the software in other hospitals.

Appropriateness of intended antibiotic prescribing using clinical case vignettes in primary care, and related factors

BACKGROUND

Factors associated with the appropriateness of antibiotic prescribing in primary care have been poorly explored. In particular, the impact of computerised decision-support systems (CDSS) remains unknown.

OBJECTIVES

We aim at investigating the uptake of CDSS and its association with physician characteristics and professional activity.

METHODS

Since May 2022, users of a CDSS for antibiotic prescribing in primary care in France have been invited, when registering, to complete three case vignettes assessing clinical situations frequently encountered in general practice and identified as at risk of antibiotic misuse. Appropriateness of antibiotic prescribing was defined as the rate of answers in line with the current guidelines, computed by individuals and by specific questions. Physician's characteristics associated with individual appropriate antibiotic prescribing (< 50%, 50-75% and > 75% appropriateness) were identified by multivariate ordinal logistic regression.

RESULTS

In June 2023, 60,067 physicians had registered on the CDSS. Among the 13,851 physicians who answered all case vignettes, the median individual appropriateness level of antibiotic prescribing was 77.8% [Interquartile range, 66.7%-88.9%], and was < 50% for 1,353 physicians (10%). In the multivariate analysis, physicians' characteristics associated with appropriateness were prior use of the CDSS (OR = 1.71, 95% CI 1.56-1.87), being a general practitioner vs. other specialist (OR = 1.34, 95% CI 1.20-1.49), working in primary care (OR = 1.14, 95% CI 1.02-1.27), mentoring students (OR = 1.12, 95% CI 1.04-1.21) age (OR = 0.69 per 10 years increase, 95% CI 0.67-0.71).

CONCLUSION

Individual appropriateness for antibiotic prescribing was high among CDSS users, with a higher rate in young general practitioners, previously using the system. CDSS could improve antibiotic prescribing in primary care.

Laboratory Informatics Approaches to Improving Care for Gender- Diverse Patients

Improving care for gender-diverse (GD) patients necessitates developing informatics tools and approaches to support optimal laboratory testing. This requires increased functionality and standardization of laboratory information system/electronic health record and data collection processes. Data tailored to accommodate immediate clinical care and clinical decision support (CDS) also have an impact on interoperability and downstream data needs for patients. Informatics tools can shape the clinical care experience for GD patients by careful design of laboratory-patient interactions.

Adaptive questionnaires for facilitating patient data entry in clinical decision support systems: methods and application to STOPP/START v2

Clinical decision support systems are software tools that help clinicians to make medical decisions. However, their acceptance by clinicians is usually rather low. A known problem is that they often require clinicians to manually enter a lot of patient data, which is long and tedious. Existing solutions, such as the automatic data extraction from electronic health record, are not fully satisfying, because of low data quality and availability. In practice, many systems still include long questionnaire for data entry. In this paper, we propose an original solution to simplify patient data entry, using an adaptive questionnaire, i.e. a questionnaire that evolves during user interaction, showing or hiding questions dynamically. Considering a rule-based decision support systems, we designed methods for determining the relationships between rules and translating the system's clinical rules into display rules that determine the items to show in the questionnaire, and methods for determining the optimal order of priority among the items in the questionnaire. We applied this approach to a decision support system implementing STOPP/START v2, a guideline for managing polypharmacy. We show that it permits reducing by about two thirds the number of clinical conditions displayed in the questionnaire, both on clinical cases and real patient data. Presented to clinicians during focus group sessions, the adaptive questionnaire was found "pretty easy to use". In the future, this approach could be applied to other guidelines, and adapted for data entry by patients.

Delivering an innovative multi-infection and female genital mutilation screening to high-risk migrant populations (ISMiHealth): study protocol of a cluster randomised controlled trial with embedded process evaluation

INTRODUCTION

ISMiHealth is a clinical decision support system, integrated as a software tool in the electronic health record system of primary care, that aims to improve the screening performance on infectious diseases and female genital mutilation (FGM) in migrants. The aim of this study is to assess the health impact of the tool and to perform a process evaluation of its feasibility and acceptability when implemented in primary care in Catalonia (Spain).

METHODS AND ANALYSIS

This study is a cluster randomised control trial where 35 primary care centres in Catalonia, Spain will be allocated into one of the two groups: intervention and control. The health professionals in the intervention centres will receive prompts, through the ISMiHealth software, with screening recommendations for infectious diseases and FGM targeting the migrant population based on an individualised risk assessment. Health professionals of the control centres will follow the current routine practice.A difference in differences analysis of the diagnostic rates for all aggregated infections and each individual condition between the intervention and control centres will be performed. Mixed-effects logistic regression models will be carried out to identify associations between the screening coverage and predictor factors. In addition, a process evaluation will be carried out using mixed methodology.

ETHICS AND DISSEMINATION

The study protocol has been approved by the institutional review boards at Hospital Clínic (16 June 2022, HCB/2022/0363), Clinical Research Ethics Committee of the Primary Care Research Institute IDIAPJGol (22 June 2022, 22/113-P) and the Almería Research Ethics Committee (27 July 2022, EMC/apg). The study will follow the tenets of the Declaration of Helsinki and Good Clinical Practice. All researchers and associates signed a collaboration agreement in which they undertake to abide by good clinical practice standards.Findings will be disseminated in peer-reviewed journals and communications to congresses.

TRIAL REGISTRATION NUMBER

NCT05868005.

Limitations of a platelet count-based clinical decision support system to facilitate diagnosis of heparin-induced thrombocytopenia

Heparin-induced thrombocytopenia (HIT) is a rare complication of heparin exposure with potential for significant morbidity and mortality. Early identification and treatment can prevent catastrophic thrombosis. Herein, we report the performance of a platelet count-based clinical decision support system (CDSS) where providers received a notification when a patient had a platelet count decline of ≥50 %. In the 90-day study period, the CDSS sent 302 notifications on 270 patients. Notifications were frequently inappropriate; 25 % had an expected platelet count decline (organ donation, stem cell transplant), an inaccurate count, or no heparin exposure. Patient testing for HIT prompted by the CDSS was not in accordance with best practice guidelines in most circumstances. For example, 36 % had a low probability 4Ts score, while 42 % with an intermediate or high probability 4Ts score were not tested. Due to concern for lack of efficacy, the CDSS was discontinued. Analysis of an 8-month period before and after discontinuation showed a significant decrease in the number of enzyme immunoassays ordered (547 vs. 386) without a change in the number of patients with HIT identified (13 vs. 13) or the rate of thrombosis in those with confirmed HIT (62 % vs. 62 %). In conclusion, a CDSS based on platelet count decline contributed to "alert fatigue" via inappropriate notification and did not improve evidence-based HIT testing. In addition, its removal did not decrease or delay HIT identification. Additional efforts are needed to better define how CDSS can support the rapid diagnosis and appropriate treatment of patients with HIT.

Climate change and artificial intelligence in healthcare: Review and recommendations towards a sustainable future

The rapid advancement of artificial intelligence (AI) in healthcare has revolutionized the industry, offering significant improvements in diagnostic accuracy, efficiency, and patient outcomes. However, the increasing adoption of AI systems also raises concerns about their environmental impact, particularly in the context of climate change. This review explores the intersection of climate change and AI in healthcare, examining the challenges posed by the energy consumption and carbon footprint of AI systems, as well as the potential solutions to mitigate their environmental impact. The review highlights the energy-intensive nature of AI model training and deployment, the contribution of data centers to greenhouse gas emissions, and the generation of electronic waste. To address these challenges, the development of energy-efficient AI models, the adoption of green computing practices, and the integration of renewable energy sources are discussed as potential solutions. The review also emphasizes the role of AI in optimizing healthcare workflows, reducing resource waste, and facilitating sustainable practices such as telemedicine. Furthermore, the importance of policy and governance frameworks, global initiatives, and collaborative efforts in promoting sustainable AI practices in healthcare is explored. The review concludes by outlining best practices for sustainable AI deployment, including eco-design, lifecycle assessment, responsible data management, and continuous monitoring and improvement. As the healthcare industry continues to embrace AI technologies, prioritizing sustainability and environmental responsibility is crucial to ensure that the benefits of AI are realized while actively contributing to the preservation of our planet.

Validation of an Electronic Health Record-Based Machine Learning Model Compared With Clinical Risk Scores for Gastrointestinal Bleeding

BACKGROUND & AIMS

Guidelines recommend use of risk stratification scores for patients presenting with gastrointestinal bleeding (GIB) to identify very-low-risk patients eligible for discharge from emergency departments. Machine learning models may outperform existing scores and can be integrated within the electronic health record (EHR) to provide real-time risk assessment without manual data entry. We present the first EHR-based machine learning model for GIB.

METHODS

The training cohort comprised 2546 patients and internal validation of 850 patients presenting with overt GIB (ie, hematemesis, melena, and hematochezia) to emergency departments of 2 hospitals from 2014 to 2019. External validation was performed on 926 patients presenting to a different hospital with the same EHR from 2014 to 2019. The primary outcome was a composite of red blood cell transfusion, hemostatic intervention (ie, endoscopic, interventional radiologic, or surgical), and 30-day all-cause mortality. We used structured data fields in the EHR, available within 4 hours of presentation, and compared the performance of machine learning models with current guideline-recommended risk scores, Glasgow-Blatchford Score, and Oakland Score. Primary analysis was area under the receiver operating characteristic curve. Secondary analysis was specificity at 99% sensitivity to assess the proportion of patients correctly identified as very low risk.

RESULTS

The machine learning model outperformed the Glasgow-Blatchford Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001) and Oakland Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001). At the very-low-risk threshold of 99% sensitivity, the machine learning model identified more very-low-risk patients: 37.9% vs 18.5% for Glasgow-Blatchford Score and 11.7% for Oakland Score (P < .001 for both comparisons).

CONCLUSIONS

An EHR-based machine learning model performs better than currently recommended clinical risk scores and identifies more very-low-risk patients eligible for discharge from the emergency department.

Clinical Considerations for Healthcare Provider-Administered Lipid-Lowering Medications

Atherosclerotic cardiovascular disease (ASCVD), a leading cause of mortality and morbidity, is associated with a substantial healthcare and economic burden. Reduction of low-density lipoprotein cholesterol (LDL-C) to guideline-recommended goals is crucial in the prevention or management of ASCVD, particularly in those at high risk. Despite the availability of several effective lipid-lowering therapies (LLTs), up to 80% of patients with ASCVD do not reach evidence-based LDL-C goals. This nonattainment may be due to poor adherence to, and lack of timely utilization of, LLTs driven by a range of variables, including polypharmacy, side effects, clinical inertia, costs, and access issues. Inclisiran was approved by the US Food and Drug Administration in 2021 as a novel, twice-yearly, healthcare provider (HCP)-administered LLT. In-office administration allows HCPs more control of drug acquisition, administration, and reimbursement, and may allow for more timely care and increased patient monitoring. In the USA, in-office administered drugs are considered a Medical Benefit and can be acquired and reimbursed using the "buy-and-bill" process. Buy-and-bill is a standard system for medication administration already established in multiple therapeutic areas, including oncology, vaccines, and allergy/immunology. Initiating in-office administration will involve new considerations for clinicians in the cardiovascular specialty, such as the implementation of new infrastructure and processes; however, it could ultimately increase treatment adherence and improve cardiovascular outcomes for patients with ASCVD. This article discusses the potential implications of buy-and-bill for the cardiology specialty and provides a practical guide to implementing HCP-administered specialty drugs in US clinical practice.

A systematic review of the value of clinical decision support systems in the prescription of antidiabetic drugs

INTRODUCTION

The management of chronic diabetes mellitus and its complications demands customized glycaemia control strategies. Polypharmacy is prevalent among people with diabetes and comorbidities, which increases the risk of adverse drug reactions. Clinical decision support systems (CDSSs) may constitute an innovative solution to these problems. The aim of our study was to conduct a systematic review assessing the value of CDSSs for the management of antidiabetic drugs (AD).

MATERIALS AND METHODS

We systematically searched the scientific literature published between January 2010 and October 2023. The retrieved studies were categorized as non-specific or AD-specific. The studies' quality was assessed using the Mixed Methods Appraisal Tool. The review's results were reported in accordance with the PRISMA guidelines.

RESULTS

Twenty studies met our inclusion criteria. The majority of AD-specific studies were conducted more recently (2020-2023) compared to non-specific studies (2010-2015). This trend hints at growing interest in more specialized CDSSs tailored for prescriptions of ADs. The nine AD-specific studies focused on metformin and insulin and demonstrated positive impacts of the CDSSs on different outcomes, including the reduction in the proportion of inappropriate prescriptions of ADs and in hypoglycaemia events. The 11 nonspecific studies showed similar trends for metformin and insulin prescriptions, although the CDSSs' impacts were not significant. There was a predominance of metformin and insulin in the studied CDSSs and a lack of studies on ADs such as sodium-glucose cotransporter-2 (SGLT-2) inhibitors and glucagon-like peptide-1 (GLP-1) receptor agonists.

CONCLUSION

The limited number of studies, especially randomized clinical trials, interested in evaluating the application of CDSS in the management of ADs underscores the need for further investigations. Our findings suggest the potential benefit of applying CDSSs to the prescription of ADs particularly in primary care settings and when targeting clinical pharmacists. Finally, establishing core outcome sets is crucial for ensuring consistent and standardized evaluation of these CDSSs.

Improving thiamine prescribing in alcohol use disorder using electronic decision support in a large urban academic medical center: A pre-post intervention study

INTRODUCTION

Thiamine is the only therapy for prevention and treatment of Wernicke Encephalopathy among patients with Alcohol Use Disorder (AUD). Despite this fact, up to 75 % of inpatients with AUD are not prescribed thiamine during hospitalization. Even fewer patients are prescribed high-dose thiamine which many experts recommend should be standard of care. Previous attempts to improve thiamine prescribing for inpatients have had limited success.

METHODS

We conducted an evaluation of thiamine prescribing in the year before and year after an intervention to increase high-dose thiamine prescribing. Pre-post study analysis occurred on two distinct study cohorts: those with alcohol-related diagnoses and those with elevated alcohol levels. The intervention was new electronic health record-based decision support which encouraged high-dose thiamine when any thiamine order was sought. No educational support was provided. The primary outcome was prescription of high-dose thiamine before versus after intervention. Of those with alcohol-related diagnoses, the monthly percentage of thiamine treatment courses including high-dose thiamine were graphed on a control chart.

RESULTS

We examined 5307 admissions with alcohol-related diagnoses (2285 pre- and 3022 post-intervention) and 698 admissions with elevated alcohol levels (319 pre- and 379 post-intervention). Among admissions with alcohol-related diagnoses, the intervention was associated with a higher proportion of admissions receiving high-dose thiamine prescriptions in the first 24 h (4.7 % vs. 1.1 %, adjusted odds ratio 4.50, CI 2.93 to 6.89, p < 0.001). A similar difference in high-dose thiamine was seen post-intervention among admissions with elevated alcohol levels (14.3 % vs. 2.5 %, adjusted odds ratio 6.43, CI 3.05 to 13.53, p < 0.001). The control chart among those with an alcohol-related diagnosis demonstrated special cause variation: the median percentage of thiamine treatment courses including high-dose thiamine improved from 8.2 % to 13.0 %.

CONCLUSIONS

Electronic decision support without educational interventions increased the use of high-dose thiamine among patients with alcohol-related diagnoses and with elevated alcohol levels during hospitalization. This increase occurred immediately in the month after the intervention and was sustained in the year-long study period after.

Exploring home healthcare clinicians' needs for using clinical decision support systems for early risk warning

OBJECTIVES

To explore home healthcare (HHC) clinicians' needs for Clinical Decision Support Systems (CDSS) information delivery for early risk warning within HHC workflows.

METHODS

Guided by the CDS "Five-Rights" framework, we conducted semi-structured interviews with multidisciplinary HHC clinicians from April 2023 to August 2023. We used deductive and inductive content analysis to investigate informants' responses regarding CDSS information delivery.

RESULTS

Interviews with thirteen HHC clinicians yielded 16 codes mapping to the CDS "Five-Rights" framework (right information, right person, right format, right channel, right time) and 11 codes for unintended consequences and training needs. Clinicians favored risk levels displayed in color-coded horizontal bars, concrete risk indicators in bullet points, and actionable instructions in the existing EHR system. They preferred non-intrusive risk alerts requiring mandatory confirmation. Clinicians anticipated risk information updates aligned with patient's condition severity and their visit pace. Additionally, they requested training to understand the CDSS's underlying logic, and raised concerns about information accuracy and data privacy.

DISCUSSION

While recognizing CDSS's value in enhancing early risk warning, clinicians highlighted concerns about increased workload, alert fatigue, and CDSS misuse. The top risk factors identified by machine learning algorithms, especially text features, can be ambiguous due to a lack of context. Future research should ensure that CDSS outputs align with clinical evidence and are explainable.

CONCLUSION

This study identified HHC clinicians' expectations, preferences, adaptations, and unintended uses of CDSS for early risk warning. Our findings endorse operationalizing the CDS "Five-Rights" framework to optimize CDSS information delivery and integration into HHC workflows.

Health Catch-UP!: a realist evaluation of an innovative multi-disease screening and vaccination tool in UK primary care for at-risk migrant patients

BACKGROUND

Migrants to the UK face disproportionate risk of infections, non-communicable diseases, and under-immunisation compounded by healthcare access barriers. Current UK migrant screening strategies are unstandardised with poor implementation and low uptake. Health Catch-UP! is a collaboratively produced digital clinical decision support system that applies current guidelines (UKHSA and NICE) to provide primary care professionals with individualised multi-disease screening (7 infectious diseases/blood-borne viruses, 3 chronic parasitic infections, 3 non-communicable disease or risk factors) and catch-up vaccination prompts for migrant patients.

METHODS

We carried out a mixed-methods process evaluation of Health Catch-UP! in two urban primary healthcare practices to integrate Health Catch-UP! into the electronic health record system of primary care, using the Medical Research Council framework for complex intervention evaluation. We collected quantitative data (demographics, patients screened, disease detection and catch-up vaccination rates) and qualitative participant interviews to explore acceptability and feasibility.

RESULTS

Ninety-nine migrants were assessed by Health Catch-UP! across two sites (S1, S2). 96.0% (n = 97) had complete demographics coding with Asia 31.3% (n = 31) and Africa 25.2% (n = 25), the most common continents of birth (S1 n = 92 [48.9% female (n = 44); mean age 60.6 years (SD 14.26)]; and S2 n = 7 [85.7% male (n = 6); mean age 39.4 years (SD16.97)]. 61.6% (n = 61) of participants were eligible for screening for at least one condition and uptake of screening was high 86.9% (n = 53). Twelve new conditions were identified (12.1% of study population) including hepatitis C (n = 1), hypercholesteraemia (n = 6), pre-diabetes (n = 4), and diabetes (n = 1). Health Catch-UP! identified that 100% (n = 99) of patients had no immunisations recorded; however, subsequent catch-up vaccination uptake was poor (2.0%, n = 1). Qualitative data supported acceptability and feasibility of Health Catch-UP! from staff and patient perspectives, and recommended Health Catch-UP! integration into routine care (e.g. NHS health checks) with an implementation package including staff and patient support materials, standardised care pathways (screening and catch-up vaccination, laboratory, and management), and financial incentivisation.

CONCLUSIONS

Clinical Decision Support Systems like Health Catch-UP! can improve disease detection and implementation of screening guidance for migrant patients but require robust testing, resourcing, and an effective implementation package to support both patients and staff.

Extubation Advisor: Implementation and Evaluation of A Novel Extubation Clinical Decision Support Tool

IMPORTANCE

Extubation Advisor (EA) is a novel software tool that generates a synoptic report for each Spontaneous Breathing Trial (SBT) conducted to inform extubation decision-making.

OBJECTIVES

To assess bedside EA implementation, perceptions of utility, and identify barriers and facilitators of use.

DESIGN, SETTING AND PARTICIPANTS

We conducted a phase I mixed-methods interventional study in three mixed intensive care unit (ICUs) in two academic hospitals. We interviewed critical care physicians (MDs) and respiratory therapists (RTs) regarding user-centered design principles and usability.

ANALYSIS

We evaluated our ability to consent participants (feasibility threshold 50%), capture complete data (threshold 90%), generate and review EA reports in real-time (thresholds 75% and 80%, respectively), and MD perception of tool usefulness (6-point Likert scale). We analyzed interview transcripts using inductive coding to identify facilitators and barriers to EA implementation and perceived benefit of tool use.

RESULTS

We enrolled 31 patients who underwent 70 SBTs. Although consent rates [31/31 (100%], complete data capture [68/68 (100%)], and EA report generation [68/70 (97.1%)] exceeded feasibility thresholds, reports were reviewed by MDs for [55/70 (78.6%)] SBTs. Mean MD usefulness score was 4.0/6. Based on feedback obtained from 36 interviews (15 MDs, 21 RTs), we revised the EA report twice and identified facilitators (ability to track patient progress, enhance extubation decision-making, and provide support in resource-limited settings) and barriers (resource constraints, need for education) to tool implementation. Half of respondents (9 MDs, 9 RTs; combined 50%) perceived definite or potential benefit to EA tool use.

CONCLUSION

This is the first study of a waveform-based variability-derived, predictive clinical decision support tool evaluated in adult ICUs. Our findings support the feasibility of integrating the EA tool into bedside workflow. Clinical trials are needed to assess the utility of the EA tool in practice and its impact on extubation decision-making and outcomes.

TRIAL REGISTRATION

NCT04708509.

Toward a Clinical Decision Support System for Monitoring Therapeutic Antituberculosis Medical Drugs in Tanzania (Project TuberXpert): Protocol for an Algorithm' Development and Implementation

BACKGROUND

The end tuberculosis (TB) strategy requires a novel patient treatment approach contrary to the one-size-fits-all model. It is well known that each patient's physiology is different and leads to various rates of drug elimination. Therapeutic drug monitoring (TDM) offers a way to manage drug dosage adaptation but requires trained pharmacologists, which is scarce in resource-limited settings.

OBJECTIVE

We will develop an automated clinical decision support system (CDSS) to help practitioners with the dosage adaptation of rifampicin, one of the essential medical drugs targeting TB, that is known for large pharmacokinetic variability and frequent suboptimal blood exposure. Such an advanced system will encourage the spread of a dosage-individualization culture, including among practitioners not specialized in pharmacology. Thus, the objectives of this project are to (1) develop the appropriate population pharmacokinetic (popPK) model for rifampicin for Tanzanian patients, (2) optimize the reporting of relevant information to practitioners for drug dosage adjustment, (3) automate the delivery of the report in line with the measurement of drug concentration, and (4) validate and implement the final system in the field.

METHODS

A total of 3 teams will combine their efforts to deliver the first automated TDM CDSS for TB. A cross-sectional study will be conducted to define the best way to display information to clinicians. In parallel, a rifampicin popPK model will be developed taking advantage of the published literature, complemented with data provided by existing literature data from the Pan-African Consortium for the Evaluation of Antituberculosis Antibiotics (panACEA), and samples collected within this project. A decision tree will be designed and implemented as a CDSS, and an automated report generation will be developed and validated through selected case studies. Expert pharmacologists will validate the CDSS, and finally, field implementation in Tanzania will occur, coupled with a prospective study to assess clinicians' adherence to the CDSS recommendations.

RESULTS

The TuberXpert project started in November 2022. In July 2024, the clinical study in Tanzania was completed with the enrollment of 50 patients to gather the required data to build a popPK model for rifampicin, together with a qualitative study defining the report design, as well as the CDSS general architecture definition.

CONCLUSIONS

At the end of the TuberXpert project, Tanzania will possess a new tool to help the practitioners with the adaptation of drug dosage targeting complicated TB cases (TB or HIV, TB or diabetes mellitus, and TB or malnutrition). This automated system will be validated and used in the field and will be proposed to other countries affected by endemic TB. In addition, this approach will serve as proof of concept regarding the feasibility and suitability of CDSS-assisted TDM for further anti-TB drugs in TB-burdened areas deprived of TDM experts, including second-line treatments considered important to monitor.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/58720.

Adoption, acceptance, and use of a decision support tool to promote timely investigations for cancer in primary care

BACKGROUND

The complexities of diagnosing cancer in general practice has driven the development of quality improvement (QI) interventions, including clinical decision support (CDS) and auditing tools. Future Health Today (FHT) is a novel QI tool, consisting of CDS at the point-of-care, practice population-level auditing, recall, and the monitoring of QI activities.

OBJECTIVES

Explore the acceptability and usability of the FHT cancer module, which flags patients with abnormal test results that may be indicative of undiagnosed cancer.

METHODS

Interviews were conducted with general practitioners (GPs) and general practice nurses (GPNs), from practices participating in a randomized trial evaluating the appropriate follow-up of patients. Clinical Performance Feedback Intervention Theory (CP-FIT) was used to analyse and interpret the data.

RESULTS

The majority of practices reported not using the auditing and QI components of the tool, only the CDS which was delivered at the point-of-care. The tool was used primarily by GPs; GPNs did not perceive the clinical recommendations to be within their role. For the CDS, facilitators for use included a good workflow fit, ease of use, low time cost, importance, and perceived knowledge gain. Barriers for use of the CDS included accuracy, competing priorities, and the patient population.

CONCLUSIONS

The CDS aligned with the clinical workflow of GPs, was considered non-disruptive to the consultation and easy to implement into usual care. By applying the CP-FIT theory, we were able to demonstrate the key drivers for GPs using the tool, and what limited the use by GPNs.

Design and development of a machine-learning-driven opioid overdose risk prediction tool integrated in electronic health records in primary care settings

BACKGROUND

Integrating advanced machine-learning (ML) algorithms into clinical practice is challenging and requires interdisciplinary collaboration to develop transparent, interpretable, and ethically sound clinical decision support (CDS) tools. We aimed to design a ML-driven CDS tool to predict opioid overdose risk and gather feedback for its integration into the University of Florida Health (UFHealth) electronic health record (EHR) system.

METHODS

We used user-centered design methods to integrate the ML algorithm into the EHR system. The backend and UI design sub-teams collaborated closely, both informed by user feedback sessions. We conducted seven user feedback sessions with five UF Health primary care physicians (PCPs) to explore aspects of CDS tools, including workflow, risk display, and risk mitigation strategies. After customizing the tool based on PCPs' feedback, we held two rounds of one-on-one usability testing sessions with 8 additional PCPs to gather feedback on prototype alerts. These sessions informed iterative UI design and backend processes, including alert frequency and reappearance circumstances.

RESULTS

The backend process development identified needs and requirements from our team, information technology, UFHealth, and PCPs. Thirteen PCPs (male = 62%, White = 85%) participated across 7 user feedback sessions and 8 usability testing sessions. During the user feedback sessions, PCPs (n = 5) identified flaws such as the term "high risk" of overdose potentially leading to unintended consequences (e.g., immediate addiction services referrals), offered suggestions, and expressed trust in the tool. In the first usability testing session, PCPs (n = 4) emphasized the need for natural risk presentation (e.g., 1 in 200) and suggested displaying the alert multiple times yearly for at-risk patients. Another 4 PCPs in the second usability testing session valued the UFHealth-specific alert for managing new or unfamiliar patients, expressed concerns about PCPs' workload when prescribing to high-risk patients, and recommended incorporating the details page into training sessions to enhance usability.

CONCLUSIONS

The final backend process for our CDS alert aligns with PCP needs and UFHealth standards. Integrating feedback from PCPs in the early development phase of our ML-driven CDS tool helped identify barriers and facilitators in the CDS integration process. This collaborative approach yielded a refined prototype aimed at minimizing unintended consequences and enhancing usability.

Applying the Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability Framework Across Implementation Stages to Identify Key Strategies to Facilitate Clinical Decision Support System Integration Within a Large Metropolitan Health Service: Interview and Focus Group Study

BACKGROUND

Computerized clinical decision support systems (CDSSs) enhance patient care through real-time, evidence-based guidance for health care professionals. Despite this, the effective implementation of these systems for health services presents multifaceted challenges, leading to inappropriate use and abandonment over the course of time. Using the Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability (NASSS) framework, this qualitative study examined CDSS adoption in a metropolitan health service, identifying determinants across implementation stages to optimize CDSS integration into health care practice.

OBJECTIVE

This study aims to identify the theory-informed (NASSS) determinants, which included multiple CDSS interventions across a 2-year period, both at the health-service level and at the individual hospital setting, that either facilitate or hinder the application of CDSSs within a metropolitan health service. In addition, this study aimed to map these determinants onto specific stages of the implementation process, thereby developing a system-level understanding of CDSS application across implementation stages.

METHODS

Participants involved in various stages of the implementation process were recruited (N=30). Participants took part in interviews and focus groups. We used a hybrid inductive-deductive qualitative content analysis and a framework mapping approach to categorize findings into barriers, enablers, or neutral determinants aligned to NASSS framework domains. These determinants were also mapped to implementation stages using the Active Implementation Framework stages approach.

RESULTS

Participants comprised clinical adopters (14/30, 47%), organizational champions (5/30, 16%), and those with roles in organizational clinical informatics (5/30, 16%). Most determinants were mapped to the organization level, technology, and adopter subdomains. However, the study findings also demonstrated a relative lack of long-term implementation planning. Consequently, determinants were not uniformly distributed across the stages of implementation, with 61.1% (77/126) identified in the exploration stage, 30.9% (39/126) in the full implementation stage, and 4.7% (6/126) in the installation stages. Stakeholders engaged in more preimplementation and full-scale implementation activities, with fewer cycles of monitoring and iteration activities identified.

CONCLUSIONS

These findings addressed a substantial knowledge gap in the literature using systems thinking principles to identify the interdependent dynamics of CDSS implementation. A lack of sustained implementation strategies (ie, training and longer-term, adopter-level championing) weakened the sociotechnical network between developers and adopters, leading to communication barriers. More rigorous implementation planning, encompassing all 4 implementation stages, may, in a way, help in addressing the barriers identified and enhancing enablers.

Views and Uses of Sepsis Digital Alerts in National Health Service Trusts in England: Qualitative Study With Health Care Professionals

BACKGROUND

Sepsis is a common cause of serious illness and death. Sepsis management remains challenging and suboptimal. To support rapid sepsis diagnosis and treatment, screening tools have been embedded into hospital digital systems to appear as digital alerts. The implementation of digital alerts to improve the management of sepsis and deterioration is a complex intervention that has to fit with team workflow and the views and practices of hospital staff. Despite the importance of human decision-making and behavior in optimal implementation, there are limited qualitative studies that explore the views and experiences of health care professionals regarding digital alerts as sepsis or deterioration computerized clinician decision support systems (CCDSSs).

OBJECTIVE

This study aims to explore the views and experiences of health care professionals on the use of sepsis or deterioration CCDSSs and to identify barriers and facilitators to their implementation and use in National Health Service (NHS) hospitals.

METHODS

We conducted a qualitative, multisite study with unstructured observations and semistructured interviews with health care professionals from emergency departments, outreach teams, and intensive or acute units in 3 NHS hospital trusts in England. Data from both interviews and observations were analyzed together inductively using thematic analysis.

RESULTS

A total of 22 health care professionals were interviewed, and 12 observation sessions were undertaken. A total of four themes regarding digital alerts were identified: (1) support decision-making as nested in electronic health records, but never substitute professionals' knowledge and experience; (2) remind to take action according to the context, such as the hospital unit and the job role; (3) improve the alerts and their introduction, by making them more accessible, easy to use, not intrusive, more accurate, as well as integrated across the whole health care system; and (4) contextual factors affecting views and use of alerts in the NHS trusts. Digital alerts are more optimally used in general hospital units with a lower senior decision maker:patient ratio and by health care professionals with experience of a similar technology. Better use of the alerts was associated with quality improvement initiatives and continuous sepsis training. The trusts' features, such as the presence of a 24/7 emergency outreach team, good technological resources, and staffing and teamwork, favored a more optimal use.

CONCLUSIONS

Trust implementation of sepsis or deterioration CCDSSs requires support on multiple levels and at all phases of the intervention, starting from a prego-live analysis addressing organizational needs and readiness. Advancements toward minimally disruptive and smart digital alerts as sepsis or deterioration CCDSSs, which are more accurate and specific but at the same time scalable and accessible, require policy changes and investments in multidisciplinary research.

Artificial intelligence (AI) applications in healthcare and considerations for nursing education

AIM/OBJECTIVE

To review the current AI applications in healthcare and explore the implications for nurse educators in innovative integration of this technology in nursing education and training programs.

BACKGROUND

There are a variety of Artificial Intelligence (AI) applications currently supporting patient care in many healthcare settings. A nursing workforce that leverages healthcare technology to enhance efficiency and accuracy of patient health outcomes is necessary. Nurse educators must understand the various uses of AI applications in healthcare to equip themselves to effectively prepare students to use the applications.

DESIGN

Qualitative synthesis and analysis of existing literature.

METHODS

Generative AI (ChatGPT) was used to support the development of a list of the current AI applications in healthcare. Each application was evaluated for relevance and accuracy. A literature review to define and understand the use of each application in clinical practice was completed. The search terms "AI" and "Health Education" were used to review the literature for evidence on educational programs used for training learners.

RESULTS

Ten current applications of AI in healthcare were identified and explored. There is little evidence that outlines how to integrate AI education into educational training for nurses.

CONCLUSION

A comprehensive multimodal educational approach that uses innovative learning strategies has potential to support the integration of AI concepts into nursing curriculum. The use of simulation and clinical practicum experiences to support experiential learning and to offer opportunities for practical application and training. Considerations for ethical use and appropriate critical evaluation of AI applications are necessary.

Clinicians' roles and necessary levels of understanding in the use of artificial intelligence: A qualitative interview study with German medical students

BACKGROUND

Artificial intelligence-driven Clinical Decision Support Systems (AI-CDSS) are being increasingly introduced into various domains of health care for diagnostic, prognostic, therapeutic and other purposes. A significant part of the discourse on ethically appropriate conditions relate to the levels of understanding and explicability needed for ensuring responsible clinical decision-making when using AI-CDSS. Empirical evidence on stakeholders' viewpoints on these issues is scarce so far. The present study complements the empirical-ethical body of research by, on the one hand, investigating the requirements for understanding and explicability in depth with regard to the rationale behind them. On the other hand, it surveys medical students at the end of their studies as stakeholders, of whom little data is available so far, but for whom AI-CDSS will be an important part of their medical practice.

METHODS

Fifteen semi-structured qualitative interviews (each lasting an average of 56 min) were conducted with German medical students to investigate their perspectives and attitudes on the use of AI-CDSS. The problem-centred interviews draw on two hypothetical case vignettes of AI-CDSS employed in nephrology and surgery. Interviewees' perceptions and convictions of their own clinical role and responsibilities in dealing with AI-CDSS were elicited as well as viewpoints on explicability as well as the necessary level of understanding and competencies needed on the clinicians' side. The qualitative data were analysed according to key principles of qualitative content analysis (Kuckartz).

RESULTS

In response to the central question about the necessary understanding of AI-CDSS tools and the emergence of their outputs as well as the reasons for the requirements placed on them, two types of argumentation could be differentiated inductively from the interviewees' statements: the first type, the clinician as a systemic trustee (or "the one relying"), highlights that there needs to be empirical evidence and adequate approval processes that guarantee minimised harm and a clinical benefit from the employment of an AI-CDSS. Based on proof of these requirements, the use of an AI-CDSS would be appropriate, as according to "the one relying", clinicians should choose those measures that statistically cause the least harm. The second type, the clinician as an individual expert (or "the one controlling"), sets higher prerequisites that go beyond ensuring empirical evidence and adequate approval processes. These higher prerequisites relate to the clinician's necessary level of competence and understanding of how a specific AI-CDSS works and how to use it properly in order to evaluate its outputs and to mitigate potential risks for the individual patient. Both types are unified in their high esteem of evidence-based clinical practice and the need to communicate with the patient on the use of medical AI. However, the interviewees' different conceptions of the clinician's role and responsibilities cause them to have different requirements regarding the clinician's understanding and explicability of an AI-CDSS beyond the proof of benefit.

CONCLUSIONS

The study results highlight two different types among (future) clinicians regarding their view of the necessary levels of understanding and competence. These findings should inform the debate on appropriate training programmes and professional standards (e.g. clinical practice guidelines) that enable the safe and effective clinical employment of AI-CDSS in various clinical fields. While current approaches search for appropriate minimum requirements of the necessary understanding and competence, the differences between (future) clinicians in terms of their information and understanding needs described here can lead to more differentiated approaches to solutions.

Using electronic health records to classify risk for adverse safety events with ICU patient Mobility: A Cross-Sectional study

BACKGROUND

Integrating early mobility (EM) expert consensus recommendations into an algorithm that uses electronic health record (EHR) data provides an opportunity for ICU nurse decision support.

OBJECTIVE

This study aimed to compare clinical differences in ICU EM algorithm domains among patients with and without documented EM and examine discordance between algorithm classification and documented EM.

METHODS

Secondary analysis of EHR data from adults admitted to the ICU from one health system's electronic data warehouse. We extracted demographic, clinical, and EM data for up to the first three days after ICU admission and applied the algorithm to classify patients as low/high-risk by clinical domain (respiratory, cardiovascular, neurological, activity order, overall) each day. We used the Wilcoxon rank sum test or Fisher's exact test to compare clinical criteria and algorithm classification between patients with and without documented EM.

RESULTS

From a total of 4,088 patients, documented EM increased each ICU day. Patients with EM were more likely to be classified by the algorithm as low-risk than those who stayed in bed each day. While a large proportion of low-risk patients remained in bed each day (813 day 1; 920 day 2; 881 day 3), some patients classified as high-risk had documented EM.

CONCLUSIONS

A significant portion of patients identified as overall low-risk by the algorithm remained in bed, while some high-risk patients achieved EM. Variability between risk definitions and documented patient activity exists and understanding additional factors that nurses use to support EM decision-making is needed.

IMPLICATIONS FOR CLINICAL PRACTICE

EHR data can be used with a mobility algorithm to classify patients at low and high-risk for ICU EM. In the future, with additional refinements, an algorithm may augment clinician decision-making.

Translation of the REMEDI[e]S (Review of potentially inappropriate MEDIcation pr[e]scribing in Seniors) explicit criteria into seminatural language for use in prescription support systems: A multidisciplinary consensus

BACKGROUND

By recovering data in an ordered manner and at the right time, clinical decision support systems (CDSSs) are designed to help healthcare professionals make decisions that improve patient care.

OBJECTIVES

The aim of the present study was to translate the REMEDI[e]s tool's explicit criteria, France's first reference list of potentially inappropriate drugs for the elderly, into seminatural language, in order to implement these criteria as alert rules and then enable their computer coding in a CDSS.

METHODS

This work was carried out at Lille University Hospital by a team of clinical pharmacists with expertise in the use of pharmaceutical decision support systems, in collaboration with the authors of the REMEDI[e]s tool. A total of 3 multi-professional consensus meetings were required to discuss the construction of each rule in seminatural language and the coding choices.

RESULTS

All REMEDIES criteria (n=104) were translated into seminatural language. This study is the first to have translated the 104 REMEDI[e]s explicit criteria into seminatural language.

CONCLUSIONS

One of the study's strengths relates to the close collaboration between the authors of the REMEDI[e]s tool and experts in CDSS programming rules; this ensured the exactitude of the seminatural language translations and limited (mis)interpretations.

Nurses' continuance intention to use electronic health record systems: The antecedent role of personality and organisation support

Nurses play a crucial role in the adoption and continued use of Electronic Health Records (EHRs), especially in developing countries. Existing literature scarcely addresses how personality traits and organisational support influence nurses' decision to persist with EHR use in these regions. This study developed a model combining the Five-Factor Model (FFM) and the Unified Theory of Acceptance and Use of Technology (UTAUT) to explore the impact of personality traits and organisational support on nurses' continuance intention to use EHR systems. Data were collected via a self-reported survey from 472 nurses across 10 public hospitals in Jordan and analyzed using a structural equation modeling approach (Smart PLS-SEM 4). The analysis revealed that personality traits, specifically Openness, Experience, and Conscientiousness, significantly influence nurses' decisions to continue using EHR systems. Furthermore, organisational support, enhanced by Performance Expectancy and Facilitating Conditions, positively affected their ongoing commitment to EHR use. The findings underscore the importance of considering individual personality traits and providing robust organisational support in promoting sustained EHR usage among nurses. These insights are vital for healthcare organisations aiming to foster a conducive environment for EHR system adoption, thereby enhancing patient care outcomes.

Comparative Study to Evaluate the Accuracy of Differential Diagnosis Lists Generated by Gemini Advanced, Gemini, and Bard for a Case Report Series Analysis: Cross-Sectional Study

BACKGROUND

Generative artificial intelligence (GAI) systems by Google have recently been updated from Bard to Gemini and Gemini Advanced as of December 2023. Gemini is a basic, free-to-use model after a user's login, while Gemini Advanced operates on a more advanced model requiring a fee-based subscription. These systems have the potential to enhance medical diagnostics. However, the impact of these updates on comprehensive diagnostic accuracy remains unknown.

OBJECTIVE

This study aimed to compare the accuracy of the differential diagnosis lists generated by Gemini Advanced, Gemini, and Bard across comprehensive medical fields using case report series.

METHODS

We identified a case report series with relevant final diagnoses published in the American Journal Case Reports from January 2022 to March 2023. After excluding nondiagnostic cases and patients aged 10 years and younger, we included the remaining case reports. After refining the case parts as case descriptions, we input the same case descriptions into Gemini Advanced, Gemini, and Bard to generate the top 10 differential diagnosis lists. In total, 2 expert physicians independently evaluated whether the final diagnosis was included in the lists and its ranking. Any discrepancies were resolved by another expert physician. Bonferroni correction was applied to adjust the P values for the number of comparisons among 3 GAI systems, setting the corrected significance level at P value <.02.

RESULTS

In total, 392 case reports were included. The inclusion rates of the final diagnosis within the top 10 differential diagnosis lists were 73% (286/392) for Gemini Advanced, 76.5% (300/392) for Gemini, and 68.6% (269/392) for Bard. The top diagnoses matched the final diagnoses in 31.6% (124/392) for Gemini Advanced, 42.6% (167/392) for Gemini, and 31.4% (123/392) for Bard. Gemini demonstrated higher diagnostic accuracy than Bard both within the top 10 differential diagnosis lists (P=.02) and as the top diagnosis (P=.001). In addition, Gemini Advanced achieved significantly lower accuracy than Gemini in identifying the most probable diagnosis (P=.002).

CONCLUSIONS

The results of this study suggest that Gemini outperformed Bard in diagnostic accuracy following the model update. However, Gemini Advanced requires further refinement to optimize its performance for future artificial intelligence-enhanced diagnostics. These findings should be interpreted cautiously and considered primarily for research purposes, as these GAI systems have not been adjusted for medical diagnostics nor approved for clinical use.

Knowledge-guided Deep Temporal Clustering for Alzheimer's Disease Subtypes in Completed Clinical Trials

Alzheimer's disease (AD) is a multifaceted neurodegenerative disorder with varied patient progression. We aim to test the hypothesis that AD patients can be categorized into subgroups based on differences in progression. We leveraged data from three randomized clinical trials (RCTs) to develop a knowledge-guided, deep temporal clustering (KG-DTC) framework for AD subtyping. This model combined autoencoders for contextual information capture, k-means clustering for representation formation, and clinical outcome classification for clinical knowledge integration. The derived representations, encompassing demographics, APOE genotype, cognitive assessments, brain volumes, and biomarkers, were clustered using the Gaussian Mixture Model to identify AD subtypes. Our novel KG-DTC framework was developed using placebo data from 2,087 AD patients across three solanezumab clinical trials (EXPEDITION, EXPEDITION2, and EXPEDITION3), achieving high performance in outcome prediction and clustering. The KG-DTC model demonstrated superior clustering structures, especially when combined with k-means clustering loss. External validation with independent clinical trial data showed consistent clustering results, with a 0.33 silhouette score for three clusters. The model's stability was confirmed through a leave-one-out approach, with an average adjusted Rand Index around 0.945. Three distinct AD subtypes were identified, each exhibiting unique patterns of cognitive function, neurodegeneration, and amyloid beta levels. Notably, Subtype 3 (S3) showed rapid cognitive decline across multiple clinical measures (e.g., 0.64 in S1 vs. -1.06 in S2 vs. 15.09 in S3 of average ADAS total change score, p<.001). This innovative approach offers promising insights for understanding variability in treatment outcomes and personalizing AD treatment strategies.

Improving Social Bot Detection Through Aid and Training

OBJECTIVE

We test the effects of three aids on individuals' ability to detect social bots among Twitter personas: a bot indicator score, a training video, and a warning.

BACKGROUND

Detecting social bots can prevent online deception. We use a simulated social media task to evaluate three aids.

METHOD

Lay participants judged whether each of 60 Twitter personas was a human or social bot in a simulated online environment, using agreement between three machine learning algorithms to estimate the probability of each persona being a bot. Experiment 1 compared a control group and two intervention groups, one provided a bot indicator score for each tweet; the other provided a warning about social bots. Experiment 2 compared a control group and two intervention groups, one receiving the bot indicator scores and the other a training video, focused on heuristics for identifying social bots.

RESULTS

The bot indicator score intervention improved predictive performance and reduced overconfidence in both experiments. The training video was also effective, although somewhat less so. The warning had no effect. Participants rarely reported willingness to share content for a persona that they labeled as a bot, even when they agreed with it.

CONCLUSIONS

Informative interventions improved social bot detection; warning alone did not.

APPLICATION

We offer an experimental testbed and methodology that can be used to evaluate and refine interventions designed to reduce vulnerability to social bots. We show the value of two interventions that could be applied in many settings.

Enhanced ICD-10 code assignment of clinical texts: A summarization-based approach

BACKGROUND

Assigning International Classification of Diseases (ICD) codes to clinical texts is a common and crucial practice in patient classification, hospital management, and further statistics analysis. Current auto-coding methods mainly transfer this task to a multi-label classification problem. Such solutions are suffering from high-dimensional mapping space and excessive redundant information in long clinical texts. To alleviate such a situation, we introduce text summarization methods to the ICD coding regime and apply text matching to select ICD codes.

METHOD

We focus on the tenth revision of the ICD (ICD-10) coding and design a novel summarization-based approach (SuM) with an end-to-end strategy to efficiently assign ICD-10 code to clinical texts. In this approach, a knowledge-guided pointer network is purposed to distill and summarize key information in clinical texts precisely. Then a matching model with matching-aggregation architecture follows to align the summary result with code, tuning the one-vs-all scenario to one-vs-one matching so that the large-label-space obstacle laid in classification approaches would be avoided.

RESULT

The 12,788 ICD-10 coded discharge summaries from a Chinese hospital were collected to evaluate the proposed approach. Compared with existing methods, the purposed model achieves the greatest coding results with Micro AUC of 0.9548, MRR@10 of 0.7977, Precision@10 of 0.0944, and Recall@10 of 0.9439 for the TOP-50 Dataset. Results on the FULL-Dataset remain consistent. Also, the proposed knowledge encoder and applied end-to-end strategy are proven to facilitate the whole model to gain efficacy in selecting the most suitable code.

CONCLUSION

The proposed automatic ICD-10 code assignment approach via text summarization can effectively capture critical messages in long clinical texts and improve the performance of ICD-10 coding of clinical texts.

Barriers and facilitators to effective electronic health record-based sepsis screening in the pediatric intensive care unit

Introduction

The Pediatric Surviving Sepsis Campaign supports the implementation of automated tools for early sepsis recognition. In 2019 the C.S. Mott Children's Hospital Pediatric Intensive Care Unit deployed an electronic medical record (EMR)-based screening for early recognition and treatment of sepsis.

Materials and Methods

We analyzed all automated primary sepsis alerts, secondary screens, and bedside huddles from November 2019 to January 2020 (Cohort 1) and from November 2020 to January 2021 (Cohort 2) to identify barriers and facilitators for the use of this tool. We distributed surveys to frontline providers to gather feedback on end-user experience.

Results

In Cohort 1, 895 primary alerts were triggered, yielding 503 completed secondary screens and 40 bedside huddles. In Cohort 2, 925 primary alerts were triggered, yielding 532 completed secondary screens and 12 bedside huddles. Surveys assessing end-user experience identified the following facilitators: (1) 73% of nurses endorsed the bedside huddle as value added; (2) 74% of medical providers agreed the bedside huddle increased the likelihood of interventions. The greatest barriers to successful implementation included the (1) overall large number of primary alerts from the automated tool and (2) rate of false alerts, many due to routine respiratory therapy interventions.

Discussion

Our data suggests that the successful implementation of EMR-based sepsis screening tools requires countermeasures focusing on 3 key drivers for change: education, technology, and patient safety.

Conclusion

While both medical providers and bedside nurses found merit in our EMR-based sepsis early recognition system, continued refinement is necessary to avoid sepsis alert fatigue.

Performance of Multimodal Artificial Intelligence Chatbots Evaluated on Clinical Oncology Cases

Importance

Multimodal artificial intelligence (AI) chatbots can process complex medical image and text-based information that may improve their accuracy as a clinical diagnostic and management tool compared with unimodal, text-only AI chatbots. However, the difference in medical accuracy of multimodal and text-only chatbots in addressing questions about clinical oncology cases remains to be tested.

Objective

To evaluate the utility of prompt engineering (zero-shot chain-of-thought) and compare the competency of multimodal and unimodal AI chatbots to generate medically accurate responses to questions about clinical oncology cases.

Design, Setting, and Participants

This cross-sectional study benchmarked the medical accuracy of multiple-choice and free-text responses generated by AI chatbots in response to 79 questions about clinical oncology cases with images.

Exposures

A unique set of 79 clinical oncology cases from JAMA Network Learning accessed on April 2, 2024, was posed to 10 AI chatbots.

Main Outcomes and Measures

The primary outcome was medical accuracy evaluated by the number of correct responses by each AI chatbot. Multiple-choice responses were marked as correct based on the ground-truth, correct answer. Free-text responses were rated by a team of oncology specialists in duplicate and marked as correct based on consensus or resolved by a review of a third oncology specialist.

Results

This study evaluated 10 chatbots, including 3 multimodal and 7 unimodal chatbots. On the multiple-choice evaluation, the top-performing chatbot was chatbot 10 (57 of 79 [72.15%]), followed by the multimodal chatbot 2 (56 of 79 [70.89%]) and chatbot 5 (54 of 79 [68.35%]). On the free-text evaluation, the top-performing chatbots were chatbot 5, chatbot 7, and the multimodal chatbot 2 (30 of 79 [37.97%]), followed by chatbot 10 (29 of 79 [36.71%]) and chatbot 8 and the multimodal chatbot 3 (25 of 79 [31.65%]). The accuracy of multimodal chatbots decreased when tested on cases with multiple images compared with questions with single images. Nine out of 10 chatbots, including all 3 multimodal chatbots, demonstrated decreased accuracy of their free-text responses compared with multiple-choice responses to questions about cancer cases.

Conclusions and Relevance

In this cross-sectional study of chatbot accuracy tested on clinical oncology cases, multimodal chatbots were not consistently more accurate than unimodal chatbots. These results suggest that further research is required to optimize multimodal chatbots to make more use of information from images to improve oncology-specific medical accuracy and reliability.

Pharmacogenomic augmented machine learning in electronic health record alerts: A health system-wide usability survey of clinicians

Pharmacogenomic (PGx) biomarkers integrated using machine learning can be embedded within the electronic health record (EHR) to provide clinicians with individualized predictions of drug treatment outcomes. Currently, however, drug alerts in the EHR are largely generic (not patient-specific) and contribute to increased clinician stress and burnout. Improving the usability of PGx alerts is an urgent need. Therefore, this work aimed to identify principles for optimal PGx alert design through a health-system-wide, mixed-methods study. Clinicians representing multiple practices and care settings (N = 1062) in urban, rural, and underserved regions were invited to complete an electronic survey comparing the usability of three drug alerts for citalopram, as a case study. Alert 1 contained a generic warning of pharmacogenomic effects on citalopram metabolism. Alerts 2 and 3 provided patient-specific predictions of citalopram efficacy with varying depth of information. Primary outcomes included the System's Usability Scale score (0-100 points) of each alert, the perceived impact of each alert on stress and decision-making, and clinicians' suggestions for alert improvement. Secondary outcomes included the assessment of alert preference by clinician age, practice type, and geographic setting. Qualitative information was captured to provide context to quantitative information. The final cohort comprised 305 geographically and clinically diverse clinicians. A simplified, individualized alert (Alert 2) was perceived as beneficial for decision-making and stress compared with a more detailed version (Alert 3) and the generic alert (Alert 1) regardless of age, practice type, or geographic setting. Findings emphasize the need for clinician-guided design of PGx alerts in the era of digital medicine.

Artificial Intelligence and Machine Learning for Inborn Errors of Immunity: Current State and Future Promise

Artificial intelligence (AI) and machine learning (ML) research within medicine has exponentially increased over the last decade, with studies showcasing the potential of AI/ML algorithms to improve clinical practice and outcomes. Ongoing research and efforts to develop AI-based models have expanded to aid in the identification of inborn errors of immunity (IEI). The use of larger electronic health record data sets, coupled with advances in phenotyping precision and enhancements in ML techniques, has the potential to significantly improve the early recognition of IEI, thereby increasing access to equitable care. In this review, we provide a comprehensive examination of AI/ML for IEI, covering the spectrum from data preprocessing for AI/ML analysis to current applications within immunology, and address the challenges associated with implementing clinical decision support systems to refine the diagnosis and management of IEI.

A clinical decision support tool for metabolic dysfunction-associated steatohepatitis in real-world clinical settings: a mixed-method implementation research study protocol

Aim: A clinical decision support (CDS) tool for metabolic dysfunction-associated steatohepatitis (MASH) was developed to align health systems with clinical guidelines detailed in the MASH Clinical Care Pathway and improve patients' proactive self-management of their disease. The tool includes a provider-facing web-based application and a mobile application (app) for patients. This protocol outlines a pilot study that will systematically evaluate the implementation of the tool in real-world clinical practice settings. Materials & methods: This implementation research study will use a simultaneous mixed-methods design and is guided by the Consolidated Framework for Implementation Research. The CDS tool for MASH will be piloted for ≥3 months at multiple US-based sites with eligible gastroenterologists and hepatologists (n = 5-10 per site) and their patients (n = 50-100 per site) with MASH or suspected MASH. Each pilot site may choose one or all focus areas within the tool (i.e., risk stratification, screening and referral, or patient care management), based on on-site capabilities. Prior to and at the end of the pilot period, providers and patients will complete quantitative surveys and partake in semi-structured interviews. Outcomes will include understanding the feasibility of implementing the tool in real-world clinical settings, its effectiveness in increasing patient screenings and risk stratification for MASH, its ability to improve provider and patient knowledge of MASH, barriers to adoption of the tool and the tool's capacity to enhance patient engagement and satisfaction with their care. Conclusion: Findings will inform the scalable implementation of the tool to ensure patients at risk for MASH are identified early, referred to specialty care when necessary and managed appropriately. Successful integration of the patient app can contribute to better health outcomes for patients by facilitating their active participation in the management of their condition.

Clinical Decision Support as a Prevention Tool for Medication Errors in the Operating Room: A Retrospective Cross-Sectional Study

BACKGROUND

Medication errors in the operating room have high potential for patient harm. While electronic clinical decision support (CDS) software has been effective in preventing medication errors in many nonoperating room patient care areas, it is not yet widely used in operating rooms. The purpose of this study was to determine the percentage of self-reported intraoperative medication errors that could be prevented by CDS algorithms.

METHODS

In this retrospective cross-sectional study, we obtained safety reports involving medication errors documented by anesthesia clinicians between August 2020 and August 2022 at a 1046-bed tertiary care academic medical center. Reviewers classified each medication error by its stage in the medication use process, error type, presence of an adverse medication event, and its associated severity and preventability by CDS. Informational gaps were corroborated by retrospective chart review and disagreements between reviewers were resolved by consensus. The primary outcome was the percentage of errors that were preventable by CDS. Secondary outcomes were preventability by CDS stratified by medication error type and severity.

RESULTS

We received 127 safety reports involving 80 medication errors, and 76/80 (95%) of the errors were classified as preventable by CDS. Certain error types were more likely to be preventable by CDS than others ( P < .001). The most likely error types to be preventable by CDS were wrong medication (N = 36, 100% rated as preventable), wrong dose (N = 30, 100% rated as preventable), and documentation errors (N = 3, 100% rated as preventable). The least likely error type to be preventable by CDS was inadvertent bolus (N = 3, none rated as preventable).

CONCLUSIONS

Ninety-five percent of self-reported medication errors in the operating room were classified as preventable by CDS. Future research should include a randomized controlled trial to assess medication error rates and types with and without the use of CDS.

Interpretable subgroup learning-based modeling framework: Study of diabetic kidney disease prediction

OBJECTIVES

Complex diseases, like diabetic kidney disease (DKD), often exhibit heterogeneity, challenging accurate risk prediction with machine learning. Traditional global models ignore patient differences, and subgroup learning lacks interpretability and predictive efficiency. This study introduces the Interpretable Subgroup Learning-based Modeling (iSLIM) framework to address these issues.

METHODS

iSLIM integrates expert knowledge with a tree-based recursive partitioning approach to identify DKD subgroups within an EHR dataset of 11,559 patients. It then constructs separate models for each subgroup, enhancing predictive accuracy while preserving interpretability.

RESULTS

Five clinically relevant subgroups are identified, achieving an average sensitivity of 0.8074, outperforming a single global model by 0.1104. Post hoc analyses provide pathological and biological evidence supporting subgroup validity and potential DKD risk factors.

CONCLUSION

The iSLIM surpasses traditional global model in predictive performance and subgroup-specific risk factor interpretation, enhancing the understanding of DKD's heterogeneous mechanisms and potentially increasing the adoption of machine learning models in clinical decision-making.

Clinician Perspectives on Decision Support and AI-based Decision Support in a Pediatric ED

BACKGROUND

Clinical decision support (CDS) systems offer the potential to improve pediatric care through enhanced test ordering, prescribing, and standardization of care. Its augmentation with artificial intelligence (AI-CDS) may help address current limitations with CDS implementation regarding alarm fatigue and accuracy of recommendations. We sought to evaluate strengths and perceptions of CDS, with a focus on AI-CDS, through semistructured interviews of clinician partners.

METHODS

We conducted a qualitative study using semistructured interviews of physicians, nurse practitioners, and nurses at a single quaternary-care pediatric emergency department to evaluate clinician perceptions of CDS and AI-CDS. We used reflexive thematic analysis to identify themes and purposive sampling to complete recruitment with the goal of reaching theoretical sufficiency.

RESULTS

We interviewed 20 clinicians. Participants demonstrated a variable understanding of CDS and AI, with some lacking a clear definition. Most recognized the potential benefits of AI-CDS in clinical contexts, such as data summarization and interpretation. Identified themes included the potential of AI-CDS to improve diagnostic accuracy, standardize care, and improve efficiency, while also providing educational benefits to clinicians. Participants raised concerns about the ability of AI-based tools to appreciate nuanced pediatric care, accurately interpret data, and about tensions between AI recommendations and clinician autonomy.

CONCLUSIONS

AI-CDS tools have a promising role in pediatric emergency medicine but require careful integration to address clinicians' concerns about autonomy, nuance recognition, and interpretability. A collaborative approach to development and implementation, informed by clinicians' insights and perspectives, will be pivotal for their successful adoption and efficacy in improving patient care.

Multiomics and eXplainable artificial intelligence for decision support in insulin resistance early diagnosis: A pediatric population-based longitudinal study

Pediatric obesity can drastically heighten the risk of cardiometabolic alterations later in life, with insulin resistance standing as the cornerstone linking adiposity to the increased cardiovascular risk. Puberty has been pointed out as a critical stage after which obesity-associated insulin resistance is more difficult to revert. Timely prediction of insulin resistance in pediatric obesity is therefore vital for mitigating the risk of its associated comorbidities. The construction of effective and robust predictive systems for a complex health outcome like insulin resistance during the early stages of life demands the adoption of longitudinal designs for more causal inferences, and the integration of factors of varying nature involved in its onset. In this work, we propose an eXplainable Artificial Intelligence-based decision support pipeline for early diagnosis of insulin resistance in a longitudinal cohort of 90 children. For that, we leverage multi-omics (genomics and epigenomics) and clinical data from the pre-pubertal stage. Different data layers combinations, pre-processing techniques (missing values, feature selection, class imbalance, etc.), algorithms, training procedures were considered following good practices for Machine Learning. SHapley Additive exPlanations were provided for specialists to understand both the decision-making mechanisms of the system and the impact of the features on each automatic decision, an essential issue in high-risk areas such as this one where system decisions may affect people's lives. The system showed a relevant predictive ability (AUC and G-mean of 0.92). A deep exploration, both at the global and the local level, revealed promising biomarkers of insulin resistance in our population, highlighting classical markers, such as Body Mass Index z-score or leptin/adiponectin ratio, and novel ones such as methylation patterns of relevant genes, such as HDAC4, PTPRN2, MATN2, RASGRF1 and EBF1. Our findings highlight the importance of integrating multi-omics data and following eXplainable Artificial Intelligence trends when building decision support systems.

Incorporating patient, caregiver, and provider perspectives in the co-design of an app to guide Hospital at Home admission decisions: a qualitative analysis

Objective

Hospital at Home (HaH) programs currently lack decision support tools to help efficiently navigate the complex decision-making process surrounding HaH as a care option. We assessed user needs and perspectives to guide early prototyping and co-creation of 4PACS (Partnering Patients and Providers for Personalized Acute Care Selection), a decision support app to help patients make an informed decision when presented with discrete hospitalization options.

Methods

From December 2021 to January 2022, we conducted semi-structured interviews via telephone with patients and caregivers recruited from Atrium Health's HaH program and physicians and a nurse with experience referring patients to HaH. Interviews were evaluated using thematic analysis. The findings were synthesized to create illustrative user descriptions to aid 4PACS development.

Results

In total, 12 stakeholders participated (3 patients, 2 caregivers, 7 providers [physicians/nurse]). We identified 4 primary themes: attitudes about HaH; 4PACS app content and information needs; barriers to 4PACS implementation; and facilitators to 4PACS implementation. We characterized 3 user descriptions (one per stakeholder group) to support 4PACS design decisions. User needs included patient selection criteria, clear program details, and descriptions of HaH components to inform care expectations. Implementation barriers included conflict between app recommendations and clinical judgement, inability to adequately represent patient-risk profile, and provider burden. Implementation facilitators included ease of use, auto-populating features, and appropriate health literacy.

Conclusions

The findings indicate important information gaps and user needs to help inform 4PACS design and barriers and facilitators to implementing 4PACS in the decision-making process of choosing between hospital-level care options.

Clinical Decision Support and Natural Language Processing in Medicine: Systematic Literature Review

BACKGROUND

Ensuring access to accurate and verified information is essential for effective patient treatment and diagnosis. Although health workers rely on the internet for clinical data, there is a need for a more streamlined approach.

OBJECTIVE

This systematic review aims to assess the current state of artificial intelligence (AI) and natural language processing (NLP) techniques in health care to identify their potential use in electronic health records and automated information searches.

METHODS

A search was conducted in the PubMed, Embase, ScienceDirect, Scopus, and Web of Science online databases for articles published between January 2000 and April 2023. The only inclusion criteria were (1) original research articles and studies on the application of AI-based medical clinical decision support using NLP techniques and (2) publications in English. A Critical Appraisal Skills Programme tool was used to assess the quality of the studies.

RESULTS

The search yielded 707 articles, from which 26 studies were included (24 original articles and 2 systematic reviews). Of the evaluated articles, 21 (81%) explained the use of NLP as a source of data collection, 18 (69%) used electronic health records as a data source, and a further 8 (31%) were based on clinical data. Only 5 (19%) of the articles showed the use of combined strategies for NLP to obtain clinical data. In total, 16 (62%) articles presented stand-alone data review algorithms. Other studies (n=9, 35%) showed that the clinical decision support system alternative was also a way of displaying the information obtained for immediate clinical use.

CONCLUSIONS

The use of NLP engines can effectively improve clinical decision systems' accuracy, while biphasic tools combining AI algorithms and human criteria may optimize clinical diagnosis and treatment flows.

TRIAL REGISTRATION

PROSPERO CRD42022373386; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=373386.

Efficacy of automated machine learning models and feature engineering for diagnosis of equivocal appendicitis using clinical and computed tomography findings

This study evaluates the diagnostic efficacy of automated machine learning (AutoGluon) with automated feature engineering and selection (autofeat), focusing on clinical manifestations, and a model integrating both clinical manifestations and CT findings in adult patients with ambiguous computed tomography (CT) results for acute appendicitis (AA). This evaluation was compared with conventional single machine learning models such as logistic regression(LR) and established scoring systems such as the Adult Appendicitis Score(AAS) to address the gap in diagnostic approaches for uncertain AA cases. In this retrospective analysis of 303 adult patients with indeterminate CT findings, the cohort was divided into appendicitis (n = 115) and non-appendicitis (n = 188) groups. AutoGluon and autofeat were used for AA prediction. The AutoGluon-clinical model relied solely on clinical data, whereas the AutoGluon-clinical-CT model included both clinical and CT data. The area under the receiver operating characteristic curve (AUROC) and other metrics for the test dataset, namely accuracy, sensitivity, specificity, PPV, NPV, and F1 score, were used to compare AutoGluon models with single machine learning models and the AAS. The single ML models in this study were LR, LASSO regression, ridge regression, support vector machine, decision tree, random forest, and extreme gradient boosting. Feature importance values were extracted using the "feature_importance" attribute from AutoGluon. The AutoGluon-clinical model demonstrated an AUROC of 0.785 (95% CI 0.691-0.890), and the ridge regression model with only clinical data revealed an AUROC of 0.755 (95% CI 0.649-0.861). The AutoGluon-clinical-CT model (AUROC 0.886 with 95% CI 0.820-0.951) performed better than the ridge model using clinical and CT data (AUROC 0.852 with 95% CI 0.774-0.930, p = 0.029). A new feature, exp(-(duration from pain to CT)3 + rebound tenderness), was identified (importance = 0.049, p = 0.001). AutoML (AutoGluon) and autoFE (autofeat) enhanced the diagnosis of uncertain AA cases, particularly when combining CT and clinical findings. This study suggests the potential of integrating AutoML and autoFE in clinical settings to improve diagnostic strategies and patient outcomes and make more efficient use of healthcare resources. Moreover, this research supports further exploration of machine learning in diagnostic processes.

A Clinical Decision Support Tool for Intimate Partner Violence Screening Among Women Veterans: Development and Qualitative Evaluation of Provider Perspectives

BACKGROUND

Women veterans, compared to civilian women, are especially at risk of experiencing intimate partner violence (IPV), pointing to the critical need for IPV screening and intervention in the Veterans Health Administration (VHA). However, implementing paper-based IPV screening and intervention in the VHA has revealed substantial barriers, including health care providers' inadequate IPV training, competing demands, time constraints, and discomfort addressing IPV and making decisions about the appropriate type or level of intervention.

OBJECTIVE

This study aimed to address IPV screening implementation barriers and hence developed and tested a novel IPV clinical decision support (CDS) tool for physicians in the Women's Health Clinic (WHC), a primary care clinic within the Veterans Affairs Palo Alto Health Care System. This tool provides intelligent, evidence-based, step-by-step guidance on how to conduct IPV screening and intervention.

METHODS

Informed by existing CDS development frameworks, developing the IPV CDS tool prototype involved six steps: (1) identifying the scope of the tool, (2) identifying IPV screening and intervention content, (3) incorporating IPV-related VHA and clinic resources, (4) identifying the tool's components, (5) designing the tool, and (6) conducting initial tool revisions. We obtained preliminary physician feedback on user experience and clinical utility of the CDS tool via the System Usability Scale (SUS) and semistructured interviews with 6 WHC physicians. SUS scores were examined using descriptive statistics. Interviews were analyzed using rapid qualitative analysis to extract actionable feedback to inform design updates and improvements.

RESULTS

This study includes a detailed description of the IPV CDS tool. Findings indicated that the tool was generally well received by physicians, who indicated good tool usability (SUS score: mean 77.5, SD 12.75). They found the tool clinically useful, needed in their practice, and feasible to implement in primary care. They emphasized that it increased their confidence in managing patients reporting IPV but expressed concerns regarding its length, workflow integration, flexibility, and specificity of information. Several physicians, for example, found the tool too time consuming when encountering patients at high risk; they suggested multiple uses of the tool (eg, an educational tool for less-experienced health care providers and a checklist for more-experienced health care providers) and including more detailed information (eg, a list of local shelters).

CONCLUSIONS

Physician feedback on the IPV CDS tool is encouraging and will be used to improve the tool. This study offers an example of an IPV CDS tool that clinics can adapt to potentially enhance the quality and efficiency of their IPV screening and intervention process. Additional research is needed to determine the tool's clinical utility in improving IPV screening and intervention rates and patient outcomes (eg, increased patient safety, reduced IPV risk, and increased referrals to mental health treatment).

Exploring the User Acceptability and Feasibility of a Clinical Decision Support Tool Designed to Facilitate Timely Diagnosis of New-Onset Type 1 Diabetes in Children: Qualitative Interview Study Among General Practitioners

BACKGROUND

Up to half of the children with new-onset type 1 diabetes present to the hospital with diabetic ketoacidosis, a life-threatening condition that can develop because of diagnostic delay. Three-quarters of Australian children visit their general practitioner (GP) the week before presenting to the hospital with diabetic ketoacidosis. Our prototype, DIRECT-T1DM (Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus), is an electronic clinical decision support tool that promotes immediate point-of-care testing in general practice to confirm the suspicion of diabetes. This avoids laboratory testing, which has been documented internationally as a cause of diagnostic delay.

OBJECTIVE

In this investigation, we aimed to pilot and assess the feasibility and acceptability of our prototype to GP end users. We also explored the challenges of diagnosing type 1 diabetes in the Australian general practice context.

METHODS

In total, 4 GPs, a pediatric endocrinologist, and a PhD candidate were involved in conceptualizing the DIRECT-T1DM prototype, which was developed at the Department of General Practice and Primary Care at the University of Melbourne. Furthermore, 6 GPs were recruited via convenience sampling to evaluate the tool. The study involved 3 phases: a presimulation interview, simulated clinical scenarios, and a postsimulation interview. The interview guide was developed using the Consolidated Framework for Implementation Research (CFIR) as a guide. All phases of the study were video, audio, and screen recorded. Audio recordings were transcribed by the investigating team. Analysis was carried out using CFIR as the underlying framework.

RESULTS

Major themes were identified among three domains and 7 constructs of the CFIR: (1) outer setting-time pressure, difficulty in diagnosing pediatric type 1 diabetes, and secondary care considerations influenced GPs' needs regarding DIRECT-T1DM; (2) inner setting-DIRECT-T1DM fits within existing workflows, it has a high relative priority due to its importance in patient safety, and GPs exhibited high tension for change; and (3) innovation-design recommendations included altering coloring to reflect urgency, font style and bolding, specific language, information and guidelines, and inclusion of patient information sheets.

CONCLUSIONS

End-user acceptability of DIRECT-T1DM was high. This was largely due to its implications for patient safety and its "real-time" nature. DIRECT-T1DM may assist in appropriate management of children with new-onset diabetes, which is an uncommon event in general practice, through safety netting.

Personalized Prediction of Long-Term Renal Function Prognosis Following Nephrectomy Using Interpretable Machine Learning Algorithms: Case-Control Study

BACKGROUND

Acute kidney injury (AKI) is a common adverse outcome following nephrectomy. The progression from AKI to acute kidney disease (AKD) and subsequently to chronic kidney disease (CKD) remains a concern; yet, the predictive mechanisms for these transitions are not fully understood. Interpretable machine learning (ML) models offer insights into how clinical features influence long-term renal function outcomes after nephrectomy, providing a more precise framework for identifying patients at risk and supporting improved clinical decision-making processes.

OBJECTIVE

This study aimed to (1) evaluate postnephrectomy rates of AKI, AKD, and CKD, analyzing long-term renal outcomes along different trajectories; (2) interpret AKD and CKD models using Shapley Additive Explanations values and Local Interpretable Model-Agnostic Explanations algorithm; and (3) develop a web-based tool for estimating AKD or CKD risk after nephrectomy.

METHODS

We conducted a retrospective cohort study involving patients who underwent nephrectomy between July 2012 and June 2019. Patient data were randomly split into training, validation, and test sets, maintaining a ratio of 76.5:8.5:15. Eight ML algorithms were used to construct predictive models for postoperative AKD and CKD. The performance of the best-performing models was assessed using various metrics. We used various Shapley Additive Explanations plots and Local Interpretable Model-Agnostic Explanations bar plots to interpret the model and generated directed acyclic graphs to explore the potential causal relationships between features. Additionally, we developed a web-based prediction tool using the top 10 features for AKD prediction and the top 5 features for CKD prediction.

RESULTS

The study cohort comprised 1559 patients. Incidence rates for AKI, AKD, and CKD were 21.7% (n=330), 15.3% (n=238), and 10.6% (n=165), respectively. Among the evaluated ML models, the Light Gradient-Boosting Machine (LightGBM) model demonstrated superior performance, with an area under the receiver operating characteristic curve of 0.97 for AKD prediction and 0.96 for CKD prediction. Performance metrics and plots highlighted the model's competence in discrimination, calibration, and clinical applicability. Operative duration, hemoglobin, blood loss, urine protein, and hematocrit were identified as the top 5 features associated with predicted AKD. Baseline estimated glomerular filtration rate, pathology, trajectories of renal function, age, and total bilirubin were the top 5 features associated with predicted CKD. Additionally, we developed a web application using the LightGBM model to estimate AKD and CKD risks.

CONCLUSIONS

An interpretable ML model effectively elucidated its decision-making process in identifying patients at risk of AKD and CKD following nephrectomy by enumerating critical features. The web-based calculator, found on the LightGBM model, can assist in formulating more personalized and evidence-based clinical strategies.

Identifying Facilitators and Barriers to Implementation of AI-Assisted Clinical Decision Support in an Electronic Health Record System

Recent advancements in computing have led to the development of artificial intelligence (AI) enabled healthcare technologies. AI-assisted clinical decision support (CDS) integrated into electronic health records (EHR) was demonstrated to have a significant potential to improve clinical care. With the rapid proliferation of AI-assisted CDS, came the realization that a lack of careful consideration of socio-technical issues surrounding the implementation and maintenance of these tools can result in unanticipated consequences, missed opportunities, and suboptimal uptake of these potentially useful technologies. The 48-h Discharge Prediction Tool (48DPT) is a new AI-assisted EHR CDS to facilitate discharge planning. This study aimed to methodologically assess the implementation of 48DPT and identify the barriers and facilitators of adoption and maintenance using the validated implementation science frameworks. The major dimensions of RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) and the constructs of the Consolidated Framework for Implementation Research (CFIR) frameworks have been used to analyze interviews of 24 key stakeholders using 48DPT. The systematic assessment of the 48DPT implementation allowed us to describe facilitators and barriers to implementation such as lack of awareness, lack of accuracy and trust, limited accessibility, and transparency. Based on our evaluation, the factors that are crucial for the successful implementation of AI-assisted EHR CDS were identified. Future implementation efforts of AI-assisted EHR CDS should engage the key clinical stakeholders in the AI tool development from the very inception of the project, support transparency and explainability of the AI models, provide ongoing education and onboarding of the clinical users, and obtain continuous input from clinical staff on the CDS performance.

Early Prediction of Cardiac Arrest in the Intensive Care Unit Using Explainable Machine Learning: Retrospective Study

BACKGROUND

Cardiac arrest (CA) is one of the leading causes of death among patients in the intensive care unit (ICU). Although many CA prediction models with high sensitivity have been developed to anticipate CA, their practical application has been challenging due to a lack of generalization and validation. Additionally, the heterogeneity among patients in different ICU subtypes has not been adequately addressed.

OBJECTIVE

This study aims to propose a clinically interpretable ensemble approach for the timely and accurate prediction of CA within 24 hours, regardless of patient heterogeneity, including variations across different populations and ICU subtypes. Additionally, we conducted patient-independent evaluations to emphasize the model's generalization performance and analyzed interpretable results that can be readily adopted by clinicians in real-time.

METHODS

Patients were retrospectively analyzed using data from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) and the eICU-Collaborative Research Database (eICU-CRD). To address the problem of underperformance, we constructed our framework using feature sets based on vital signs, multiresolution statistical analysis, and the Gini index, with a 12-hour window to capture the unique characteristics of CA. We extracted 3 types of features from each database to compare the performance of CA prediction between high-risk patient groups from MIMIC-IV and patients without CA from eICU-CRD. After feature extraction, we developed a tabular network (TabNet) model using feature screening with cost-sensitive learning. To assess real-time CA prediction performance, we used 10-fold leave-one-patient-out cross-validation and a cross-data set method. We evaluated MIMIC-IV and eICU-CRD across different cohort populations and subtypes of ICU within each database. Finally, external validation using the eICU-CRD and MIMIC-IV databases was conducted to assess the model's generalization ability. The decision mask of the proposed method was used to capture the interpretability of the model.

RESULTS

The proposed method outperformed conventional approaches across different cohort populations in both MIMIC-IV and eICU-CRD. Additionally, it achieved higher accuracy than baseline models for various ICU subtypes within both databases. The interpretable prediction results can enhance clinicians' understanding of CA prediction by serving as a statistical comparison between non-CA and CA groups. Next, we tested the eICU-CRD and MIMIC-IV data sets using models trained on MIMIC-IV and eICU-CRD, respectively, to evaluate generalization ability. The results demonstrated superior performance compared with baseline models.

CONCLUSIONS

Our novel framework for learning unique features provides stable predictive power across different ICU environments. Most of the interpretable global information reveals statistical differences between CA and non-CA groups, demonstrating its utility as an indicator for clinical decisions. Consequently, the proposed CA prediction system is a clinically validated algorithm that enables clinicians to intervene early based on CA prediction information and can be applied to clinical trials in digital health.

Intervention skills - a neglected field of research in medical education and beyond

Intervention reasoning as a critical component of clinical reasoning has been understudied in medical education in contrast to the well-established field of diagnostic reasoning. This resonates in a lack of comprehensive understanding of the cognitive processes involved and a deficit in research to promote intervention skills in future clinicians. In this commentary, we present a conceptual framework for intervention reasoning that includes four phases: generating, selecting, implementing, and evaluating interventions. The conceptualization highlights cognitive processes such as developing interventions based on a patient's diagnosis and signs and symptoms; selecting the most appropriate option by contrasting, prioritizing, and evaluating interventions in terms of feasibility, effectiveness, and the patient's context-specific needs; and predicting patient outcomes within so-called "developmental corridors" to adjust treatments accordingly. In addition to these cognitive processes, interventions require collaborative activities, such as sharing information with other care providers, distributing roles among care teams, or acting together. Future research should validate the proposed framework, examine the impact of intervention reasoning on clinical outcomes, and identify effective training methods (e.g., simulation and AI-based approaches). In addition, it would be valuable to explore the transferability and generalizability of the model to other areas of health education and contexts outside of health education.