Machine Learning Approach to Reduce Alert Fatigue Using a Disease Medication-Related Clinical Decision Support System: Model Development and Validation

The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review

OBJECTIVE

Current Clinical Decision Support Systems (CDSSs) generate medication alerts that are of limited clinical value, causing alert fatigue. Artificial Intelligence (AI)-based methods may help in optimizing medication alerts. Therefore, we conducted a scoping review on the current state of the use of AI to optimize medication alerts in a hospital setting. Specifically, we aimed to identify the applied AI methods used together with their performance measures and main outcome measures.

MATERIALS AND METHODS

We searched Medline, Embase, and Cochrane Library database on May 25, 2023 for studies of any quantitative design, in which the use of AI-based methods was investigated to optimize medication alerts generated by CDSSs in a hospital setting. The screening process was supported by ASReview software.

RESULTS

Out of 5625 citations screened for eligibility, 10 studies were included. Three studies (30%) reported on both statistical performance and clinical outcomes. The most often reported performance measure was positive predictive value ranging from 9% to 100%. Regarding main outcome measures, alerts optimized using AI-based methods resulted in a decreased alert burden, increased identification of inappropriate or atypical prescriptions, and enabled prediction of user responses. In only 2 studies the AI-based alerts were implemented in hospital practice, and none of the studies conducted external validation.

DISCUSSION AND CONCLUSION

AI-based methods can be used to optimize medication alerts in a hospital setting. However, reporting on models' development and validation should be improved, and external validation and implementation in hospital practice should be encouraged.

Potential Applications of Artificial Intelligence (AI) in Managing Polypharmacy in Saudi Arabia: A Narrative Review

Prescribing medications is a fundamental practice in the management of illnesses that necessitates in-depth knowledge of clinical pharmacology. Polypharmacy, or the concurrent use of multiple medications by individuals with complex health conditions, poses significant challenges, including an increased risk of drug interactions and adverse reactions. The Saudi Vision 2030 prioritises enhancing healthcare quality and safety, including addressing polypharmacy. Artificial intelligence (AI) offers promising tools to optimise medication plans, predict adverse drug reactions and ensure drug safety. This review explores AI's potential to revolutionise polypharmacy management in Saudi Arabia, highlighting practical applications, challenges and the path forward for the integration of AI solutions into healthcare practices.

Ability of machine-learning based clinical decision support system to reduce alert fatigue, wrong-drug errors, and alert users about look alike, sound alike medication

BACKGROUND AND OBJECTIVE

The overall benefits of using clinical decision support systems (CDSSs) can be restrained if physicians inadvertently ignore clinically useful alerts due to "alert fatigue" caused by an excessive number of clinically irrelevant warnings. Moreover, inappropriate drug errors, look-alike/sound-alike (LASA) drug errors, and problem list documentation are common, costly, and potentially harmful. This study sought to evaluate the overall performance of a machine learning-based CDSS (MedGuard) for triggering clinically relevant alerts, acceptance rate, and to intercept inappropriate drug errors as well as LASA drug errors.

METHODS

We conducted a retrospective study that evaluated MedGuard alerts, the alert acceptance rate, and the rate of LASA alerts between July 1, 2019, and June 31, 2021, from outpatient settings at an academic hospital. An expert pharmacist checked the suitability of the alerts, rate of acceptance, wrong-drug errors, and confusing drug pairs.

RESULTS

Over the two-year study period, 1,206,895 prescriptions were ordered and a total of 28,536 alerts were triggered (alert rate: 2.36 %). Of the 28,536 alerts presented to physicians, 13,947 (48.88 %) were accepted. A total of 8,014 prescriptions were changed/modified (28.08 %, 8,014/28,534) with the most common reasons being adding and/or deleting diseases (52.04 %, 4,171/8,014), adding and/or deleting drugs (21.89 %, 1,755/8,014) and others (35.48 %, 2,844/ 8,014). However, the rate of drug error interception was 1.64 % (470 intercepted errors out of 28,536 alerts), which equates to 16.4 intercepted errors per 1000 alerted orders.

CONCLUSION

This study shows that machine learning based CDSS, MedGuard, has an ability to improve patients' safety by triggering clinically valid alerts. This system can also help improve problem list documentation and intercept inappropriate drug errors and LASA drug errors, which can improve medication safety. Moreover, high acceptance of alert rates can help reduce clinician burnout and adverse events.

Physicians' Perspectives on AI in Clinical Decision Support Systems: Interview Study of the CURATE.AI Personalized Dose Optimization Platform

BACKGROUND

Physicians play a key role in integrating new clinical technology into care practices through user feedback and growth propositions to developers of the technology. As physicians are stakeholders involved through the technology iteration process, understanding their roles as users can provide nuanced insights into the workings of these technologies that are being explored. Therefore, understanding physicians' perceptions can be critical toward clinical validation, implementation, and downstream adoption. Given the increasing prevalence of clinical decision support systems (CDSSs), there remains a need to gain an in-depth understanding of physicians' perceptions and expectations toward their downstream implementation. This paper explores physicians' perceptions of integrating CURATE.AI, a novel artificial intelligence (AI)-based and clinical stage personalized dosing CDSSs, into clinical practice.

OBJECTIVE

This study aims to understand physicians' perspectives of integrating CURATE.AI for clinical work and to gather insights on considerations of the implementation of AI-based CDSS tools.

METHODS

A total of 12 participants completed semistructured interviews examining their knowledge, experience, attitudes, risks, and future course of the personalized combination therapy dosing platform, CURATE.AI. Interviews were audio recorded, transcribed verbatim, and coded manually. The data were thematically analyzed.

RESULTS

Overall, 3 broad themes and 9 subthemes were identified through thematic analysis. The themes covered considerations that physicians perceived as significant across various stages of new technology development, including trial, clinical implementation, and mass adoption.

CONCLUSIONS

The study laid out the various ways physicians interpreted an AI-based personalized dosing CDSS, CURATE.AI, for their clinical practice. The research pointed out that physicians' expectations during the different stages of technology exploration can be nuanced and layered with expectations of implementation that are relevant for technology developers and researchers.

Development and validation of a machine learning-based detection system to improve precision screening for medication errors in the neonatal intensive care unit

Aim: To develop models that predict the presence of medication errors (MEs) (prescription, preparation, administration, and monitoring) using machine learning in NICU patients. Design: Prospective, observational cohort study randomized with machine learning (ML) algorithms. Setting: A 22-bed capacity NICU in Ankara, Turkey, between February 2020 and July 2021. Results: A total of 11,908 medication orders (28.9 orders/patient) for 412 NICU patients (5.53 drugs/patient/day) who received 2,280 prescriptions over 32,925 patient days were analyzed. At least one physician-related ME and nurse-related ME were found in 174 (42.2%) and 235 (57.0%) of the patients, respectively. The parameters that had the highest correlation with ME occurrence and subsequently included in the model were: total number of drugs, anti-infective drugs, nervous system drugs, 5-min APGAR score, postnatal age, alimentary tract and metabolism drugs, and respiratory system drugs as patient-related parameters, and weekly working hours of nurses, weekly working hours of physicians, and number of nurses' monthly shifts as care provider-related parameters. The obtained model showed high performance to predict ME (AUC: 0.920; 95% CI: 0.876-0.970) presence and is accessible online (http://softmed.hacettepe.edu.tr/NEO-DEER_Medication_Error/). Conclusion: This is the first developed and validated model to predict the presence of ME using work environment and pharmacotherapy parameters with high-performance ML algorithms in NICU patients. This approach and the current model hold the promise of implementation of targeted/precision screening to prevent MEs in neonates. Clinical Trial Registration: ClinicalTrials.gov, identifier NCT04899960.

Development and Validation of Novel Deep-Learning Models Using Multiple Data Types for Lung Cancer Survival

A well-established lung-cancer-survival-prediction model that relies on multiple data types, multiple novel machine-learning algorithms, and external testing is absent in the literature. This study aims to address this gap and determine the critical factors of lung cancer survival. We selected non-small-cell lung cancer patients from a retrospective dataset of the Taipei Medical University Clinical Research Database and Taiwan Cancer Registry between January 2008 and December 2018. All patients were monitored from the index date of cancer diagnosis until the event of death. Variables, including demographics, comorbidities, medications, laboratories, and patient gene tests, were used. Nine machine-learning algorithms with various modes were used. The performance of the algorithms was measured by the area under the receiver operating characteristic curve (AUC). In total, 3714 patients were included. The best performance of the artificial neural network (ANN) model was achieved when integrating all variables with the AUC, accuracy, precision, recall, and F1-score of 0.89, 0.82, 0.91, 0.75, and 0.65, respectively. The most important features were cancer stage, cancer size, age of diagnosis, smoking, drinking status, EGFR gene, and body mass index. Overall, the ANN model improved predictive performance when integrating different data types.

Appropriateness of Alerts and Physicians’ Responses with a Medication-related Clinical Decision Support System (Preprint)

Background

Alert fatigue is unavoidable when many irrelevant alerts are generated in response to a small number of useful alerts. It is necessary to increase the effectiveness of the clinical decision support system (CDSS) by understanding physicians’ responses.

Objective

This study aimed to understand the CDSS and physicians’ behavior by evaluating the clinical appropriateness of alerts and the corresponding physicians’ responses in a medication-related passive alert system.

Methods

Data on medication-related orders, alerts, and patients’ electronic medical records were analyzed. The analyzed data were generated between August 2019 and June 2020 while the patient was in the emergency department. We evaluated the appropriateness of alerts and physicians’ responses for a subset of 382 alert cases and classified them.

Results

Of the 382 alert cases, only 7.3% (n=28) of the alerts were clinically appropriate. Regarding the appropriateness of the physicians’ responses about the alerts, 92.4% (n=353) were deemed appropriate. In the classification of alerts, only 3.4% (n=13) of alerts were successfully triggered, and 2.1% (n=8) were inappropriate in both alert clinical relevance and physician’s response. In this study, the override rate was 92.9% (n=355).

Conclusions

We evaluated the appropriateness of alerts and physicians’ responses through a detailed medical record review of the medication-related passive alert system. An excessive number of unnecessary alerts are generated, because the algorithm operates as a rule base without reflecting the individual condition of the patient. It is important to maximize the value of the CDSS by comprehending physicians’ responses.

The potential for leveraging machine learning to filter medication alerts

OBJECTIVE

To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view.

MATERIALS AND METHODS

We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision.

RESULTS

A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001).

CONCLUSIONS

Machine learning potentially enables the intelligent filtering of medication alerts.

Clinician Decisions After Notification of Elevated Blood Pressure Measurements From Patients in a Remote Monitoring Program

IMPORTANCE

Guidelines recommend using telehealth for hypertension management, but insufficient evidence is available to guide strategies for incorporating telehealth data into clinical practice.

OBJECTIVE

To describe how primary care teams responded to elevated remote blood pressure (BP) alerts in the electronic health record (EHR) in a randomized clinical trial of BP telemonitoring conducted in routine practice settings.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study reviewed EHR documentation from May 8, 2018, to August 9, 2019, in a single urban academic family practice site. Primary care teams comprising 28 attending physicians and nurse practitioners, residents, and nurses cared for 162 patients in a text-based clinical trial of remote BP monitoring remote BP monitoring. Data were analyzed from October 21, 2019, to April 30, 2021.

EXPOSURES

Clinicians received a direct message in their EHR inbox when patients submitted at least 3 elevated BP readings.

MAIN OUTCOMES AND MEASURES

Categories and frequencies of clinician action, created via review of EHR-documented clinician responses to EHR alerts by 2 physicians.

RESULTS

Patients in this study (n = 162) were predominantly female (111 [68.5%]) and Black or African American (146 [90.1%]), whereas attending physicians (n = 21) were predominantly female (13 [61.9%]) and non-Hispanic White (19 [90.5%]) with a mean (SD) age of 51.6 (11.1) years. Five hundred fifty-two alerts fell into 12 categories of clinical actions. Clinicians acted on 343 alerts (62.1%). Common remote activities were to reconcile medications and assess adherence (120 of 552 alerts [21.7%]) and verify BP measurement technique (65 of 552 alerts [11.8%]). Clinicians also commonly requested appointments (120 of 552 alerts [21.7%]) and/or saw the patient in a subsequent office visit (114 of 552 alerts [20.7%]). Ninety-six alerts (17.4%) resulted in medication changes; half of these changes were remote (48 of 96 [50.0%]), and the other half were visit-based. For 209 of 552 alerts (37.9%), no changes were made to the care plan, typically without documenting clinical rationale (196 of 209 instances [93.8%]). Exploratory EHR review was used to infer potential clinical rationale for 106 (54.1%) of such cases, but there was insufficient information for the remaining 90 (45.9%).

CONCLUSIONS AND RELEVANCE

These findings suggest that EHR alerts for elevated BP during remote monitoring were effective in prompting a mix of remote and office-based management. It was also common for the plan of care to remain unchanged, possibly suggesting need for more refined alerts and improved clinician support.

Physician Knowledge Base: Clinical Decision Support Systems

With the introduction of electronic medical records (EMRs), it has become possible to accumulate massive amounts of qualitative medical data. As such, EMRs have become increasingly used in clinical decision support systems (CDSSs). While CDSSs aim to reduce medical errors normally occurring in the process of treating patients by physicians, technical maturity and the completeness of CDSSs do not meet standards for medical use yet. As data further accumulates, CDSS algorithms must be continuously updated to allow CDSSs to perform their core functions. Doing so, however, requires extensive time and manpower investments. In current practice, computational systems already perform a wide variety of functions in medical settings to allow medical staff to focus on other tasks. However, no prior research has evaluated the potential effectiveness of future CDSSs nor analyzed possibilities for their further development. In this article, we evaluate CDSS technology with the consideration that medical staff also understand the core functions of such systems.

Clinical Decision Support System Based on Hybrid Knowledge Modeling: A Case Study of Chronic Kidney Disease-Mineral and Bone Disorder Treatment

Clinical decision support systems (CDSSs) represent the latest technological transformation in healthcare for assisting clinicians in complex decision-making. Several CDSSs are proposed to deal with a range of clinical tasks such as disease diagnosis, prescription management, and medication ordering. Although a small number of CDSSs have focused on treatment selection, areas such as medication selection and dosing selection remained under-researched. In this regard, this study represents one of the first studies in which a CDSS is proposed for clinicians who manage patients with end-stage renal disease undergoing maintenance hemodialysis, almost all of whom have some manifestation of chronic kidney disease-mineral and bone disorder (CKD-MBD). The primary objective of the system is to aid clinicians in dosage prescription by levering medical domain knowledge as well existing practices. The proposed CDSS is evaluated with a real-world hemodialysis patient dataset acquired from Kyung Hee University Hospital, South Korea. Our evaluation demonstrates overall high compliance based on the concordance metric between the proposed CKD-MBD CDSS recommendations and the routine clinical practice. The concordance rate of overall medication dosing selection is 78.27%. Furthermore, the usability aspects of the system are also evaluated through the User Experience Questionnaire method to highlight the appealing aspects of the system for clinicians. The overall user experience dimension scores for pragmatic, hedonic, and attractiveness are 1.53, 1.48, and 1.41, respectively. A service reliability for the Cronbach's alpha coefficient greater than 0.7 is achieved using the proposed system, whereas a dependability coefficient of the value 0.84 reveals a significant effect.

Application of Artificial Intelligence in COVID-19 Pandemic: Bibliometric Analysis

The application of artificial intelligence (AI) to health has increased, including to COVID-19. This study aimed to provide a clear overview of COVID-19-related AI publication trends using longitudinal bibliometric analysis. A systematic literature search was conducted on the Web of Science for English language peer-reviewed articles related to AI application to COVID-19. A search strategy was developed to collect relevant articles and extracted bibliographic information (e.g., country, research area, sources, and author). VOSviewer (Leiden University) and Bibliometrix (R package) were used to visualize the co-occurrence networks of authors, sources, countries, institutions, global collaborations, citations, co-citations, and keywords. We included 729 research articles on the application of AI to COVID-19 published between 2020 and 2021. PLOS One (33/729, 4.52%), Chaos Solution Fractals (29/729, 3.97%), and Journal of Medical Internet Research (29/729, 3.97%) were the most common journals publishing these articles. The Republic of China (190/729, 26.06%), the USA (173/729, 23.73%), and India (92/729, 12.62%) were the most prolific countries of origin. The Huazhong University of Science and Technology, Wuhan University, and the Chinese Academy of Sciences were the most productive institutions. This is the first study to show a comprehensive picture of the global efforts to address COVID-19 using AI. The findings of this study also provide insights and research directions for academic researchers, policymakers, and healthcare practitioners who wish to collaborate in these domains in the future.