A roadmap for national action on clinical decision support

How intervention studies measure the effectiveness of medication safety-related clinical decision support systems in primary and long-term care: a systematic review

BACKGROUND

Medication errors and associated adverse drug events (ADE) are a major cause of morbidity and mortality worldwide. In recent years, the prevention of medication errors has become a high priority in healthcare systems. In order to improve medication safety, computerized Clinical Decision Support Systems (CDSS) are increasingly being integrated into the medication process. Accordingly, a growing number of studies have investigated the medication safety-related effectiveness of CDSS. However, the outcome measures used are heterogeneous, leading to unclear evidence. The primary aim of this study is to summarize and categorize the outcomes used in interventional studies evaluating the effects of CDSS on medication safety in primary and long-term care.

METHODS

We systematically searched PubMed, Embase, CINAHL, and Cochrane Library for interventional studies evaluating the effects of CDSS targeting medication safety and patient-related outcomes. We extracted methodological characteristics, outcomes and empirical findings from the included studies. Outcomes were assigned to three main categories: process-related, harm-related, and cost-related. Risk of bias was assessed using the Evidence Project risk of bias tool.

RESULTS

Thirty-two studies met the inclusion criteria. Almost all studies (n = 31) used process-related outcomes, followed by harm-related outcomes (n = 11). Only three studies used cost-related outcomes. Most studies used outcomes from only one category and no study used outcomes from all three categories. The definition and operationalization of outcomes varied widely between the included studies, even within outcome categories. Overall, evidence on CDSS effectiveness was mixed. A significant intervention effect was demonstrated by nine of fifteen studies with process-related primary outcomes (60%) but only one out of five studies with harm-related primary outcomes (20%). The included studies faced a number of methodological problems that limit the comparability and generalizability of their results.

CONCLUSIONS

Evidence on the effectiveness of CDSS is currently inconclusive due in part to inconsistent outcome definitions and methodological problems in the literature. Additional high-quality studies are therefore needed to provide a comprehensive account of CDSS effectiveness. These studies should follow established methodological guidelines and recommendations and use a comprehensive set of harm-, process- and cost-related outcomes with agreed-upon and consistent definitions.

PROSPERO REGISTRATION

CRD42023464746.

Improving Naloxone Co-prescribing Through Clinical Decision Support

BACKGROUND

Despite national guidelines recommending naloxone co-prescription with high-risk medications, rates remain low nationally. This was reflected at our institution with remarkably low naloxone prescribing rates. We sought to determine if a clinical decision support (CDS) tool could increase rates of naloxone co-prescribing with high-risk prescriptions.

METHODS

 An alert in the electronic health record was triggered upon signing an order for a high-risk opioid medication without a naloxone co-prescription. We examined all opioid prescriptions written by family and general internal medicine practitioners at the University of Iowa Hospitals and Clinics in outpatient encounters between November 30, 2020, and February 28, 2022. Once triggered by a high-risk prescription, the CDS tool had the option to choose an order set with an automatically selected co-prescription for naloxone along with patient instructions automatically added to the patient's after-visit summary (AVS). We examined the monthly percentage of patients receiving Schedule II opioid prescriptions ≥90 morphine milliequivalents (MME)/day who received concurrent naloxone prescriptions in the 12 months before the CDS went live and the three months following go-live.

RESULTS

 Concurrent naloxone prescriptions increased from 1.1% in the 12 months prior to implementation in November 2021 to 9.4% (p<0.001) during the post-intervention period across eight family medicine and internal medicine clinics.

DISCUSSION

 This single-center quality improvement project with retrospective analysis demonstrates the potential efficacy of a single CDS tool in increasing the rate of naloxone prescription. The impact of such prescribing on overall mortality requires further research.

CONCLUSIONS

The CDS tool was easy to implement and improved rates of appropriate naloxone co-prescribing.

What Are Humans Doing in the Loop? Co-Reasoning and Practical Judgment When Using Machine Learning-Driven Decision Aids

Within the ethical debate on Machine Learning-driven decision support systems (ML_CDSS), notions such as "human in the loop" or "meaningful human control" are often cited as being necessary for ethical legitimacy. In addition, ethical principles usually serve as the major point of reference in ethical guidance documents, stating that conflicts between principles need to be weighed and balanced against each other. Starting from a neo-Kantian viewpoint inspired by Onora O'Neill, this article makes a concrete suggestion of how to interpret the role of the "human in the loop" and to overcome the perspective of rivaling ethical principles in the evaluation of AI in health care. We argue that patients should be perceived as "fellow workers" and epistemic partners in the interpretation of ML_CDSS outputs. We further highlight that a meaningful process of integrating (rather than weighing and balancing) ethical principles is most appropriate in the evaluation of medical AI.

Clinical Decision Support for Hypertension Management in Chronic Kidney Disease: A Randomized Clinical Trial

Importance

Chronic kidney disease (CKD) affects 37 million adults in the United States, and for patients with CKD, hypertension is a key risk factor for adverse outcomes, such as kidney failure, cardiovascular events, and death.

Objective

To evaluate a computerized clinical decision support (CDS) system for the management of uncontrolled hypertension in patients with CKD.

Design, Setting, and Participants

This multiclinic, randomized clinical trial randomized primary care practitioners (PCPs) at a primary care network, including 15 hospital-based, ambulatory, and community health center-based clinics, through a stratified, matched-pair randomization approach February 2021 to February 2022. All adult patients with a visit to a PCP in the last 2 years were eligible and those with evidence of CKD and hypertension were included.

Intervention

The intervention consisted of a CDS system based on behavioral economic principles and human-centered design methods that delivered tailored, evidence-based recommendations, including initiation or titration of renin-angiotensin-aldosterone system inhibitors. The patients in the control group received usual care from PCPs with the CDS system operating in silent mode.

Main Outcomes and Measures

The primary outcome was the change in mean systolic blood pressure (SBP) between baseline and 180 days compared between groups. The primary analysis was a repeated measures linear mixed model, using SBP at baseline, 90 days, and 180 days in an intention-to-treat repeated measures model to account for missing data. Secondary outcomes included blood pressure (BP) control and outcomes such as percentage of patients who received an action that aligned with the CDS recommendations.

Results

The study included 174 PCPs and 2026 patients (mean [SD] age, 75.3 [0.3] years; 1223 [60.4%] female; mean [SD] SBP at baseline, 154.0 [14.3] mm Hg), with 87 PCPs and 1029 patients randomized to the intervention and 87 PCPs and 997 patients randomized to usual care. Overall, 1714 patients (84.6%) were treated for hypertension at baseline. There were 1623 patients (80.1%) with an SBP measurement at 180 days. From the linear mixed model, there was a statistically significant difference in mean SBP change in the intervention group compared with the usual care group (change, -14.6 [95% CI, -13.1 to -16.0] mm Hg vs -11.7 [-10.2 to -13.1] mm Hg; P = .005). There was no difference in the percentage of patients who achieved BP control in the intervention group compared with the control group (50.4% [95% CI, 46.5% to 54.3%] vs 47.1% [95% CI, 43.3% to 51.0%]). More patients received an action aligned with the CDS recommendations in the intervention group than in the usual care group (49.9% [95% CI, 45.1% to 54.8%] vs 34.6% [95% CI, 29.8% to 39.4%]; P < .001).

Conclusions and Relevance

These findings suggest that implementing this computerized CDS system could lead to improved management of uncontrolled hypertension and potentially improved clinical outcomes at the population level for patients with CKD.

Trial Registration

ClinicalTrials.gov Identifier: NCT03679247.

Genetically guided precision medicine clinical decision support tools: a systematic review

OBJECTIVES

Patient care using genetics presents complex challenges. Clinical decision support (CDS) tools are a potential solution because they provide patient-specific risk assessments and/or recommendations at the point of care. This systematic review evaluated the literature on CDS systems which have been implemented to support genetically guided precision medicine (GPM).

MATERIALS AND METHODS

A comprehensive search was conducted in MEDLINE and Embase, encompassing January 1, 2011-March 14, 2023. The review included primary English peer-reviewed research articles studying humans, focused on the use of computers to guide clinical decision-making and delivering genetically guided, patient-specific assessments, and/or recommendations to healthcare providers and/or patients.

RESULTS

The search yielded 3832 unique articles. After screening, 41 articles were identified that met the inclusion criteria. Alerts and reminders were the most common form of CDS used. About 27 systems were integrated with the electronic health record; 2 of those used standards-based approaches for genomic data transfer. Three studies used a framework to analyze the implementation strategy.

DISCUSSION

Findings include limited use of standards-based approaches for genomic data transfer, system evaluations that do not employ formal frameworks, and inconsistencies in the methodologies used to assess genetic CDS systems and their impact on patient outcomes.

CONCLUSION

We recommend that future research on CDS system implementation for genetically GPM should focus on implementing more CDS systems, utilization of standards-based approaches, user-centered design, exploration of alternative forms of CDS interventions, and use of formal frameworks to systematically evaluate genetic CDS systems and their effects on patient care.

Clinical Decision Support Requirements for Ventricular Tachycardia Diagnosis Within the Frameworks of Knowledge and Practice: Survey Study

BACKGROUND

Ventricular tachycardia (VT) diagnosis is challenging due to the similarity between VT and some forms of supraventricular tachycardia, complexity of clinical manifestations, heterogeneity of underlying diseases, and potential for life-threatening hemodynamic instability. Clinical decision support systems (CDSSs) have emerged as promising tools to augment the diagnostic capabilities of cardiologists. However, a requirements analysis is acknowledged to be vital for the success of a CDSS, especially for complex clinical tasks such as VT diagnosis.

OBJECTIVE

The aims of this study were to analyze the requirements for a VT diagnosis CDSS within the frameworks of knowledge and practice and to determine the clinical decision support (CDS) needs.

METHODS

Our multidisciplinary team first conducted semistructured interviews with seven cardiologists related to the clinical challenges of VT and expected decision support. A questionnaire was designed by the multidisciplinary team based on the results of interviews. The questionnaire was divided into four sections: demographic information, knowledge assessment, practice assessment, and CDS needs. The practice section consisted of two simulated cases for a total score of 10 marks. Online questionnaires were disseminated to registered cardiologists across China from December 2022 to February 2023. The scores for the practice section were summarized as continuous variables, using the mean, median, and range. The knowledge and CDS needs sections were assessed using a 4-point Likert scale without a neutral option. Kruskal-Wallis tests were performed to investigate the relationship between scores and practice years or specialty.

RESULTS

Of the 687 cardiologists who completed the questionnaire, 567 responses were eligible for further analysis. The results of the knowledge assessment showed that 383 cardiologists (68%) lacked knowledge in diagnostic evaluation. The overall average score of the practice assessment was 6.11 (SD 0.55); the etiological diagnosis section had the highest overall scores (mean 6.74, SD 1.75), whereas the diagnostic evaluation section had the lowest scores (mean 5.78, SD 1.19). A majority of cardiologists (344/567, 60.7%) reported the need for a CDSS. There was a significant difference in practice competency scores between general cardiologists and arrhythmia specialists (P=.02).

CONCLUSIONS

There was a notable deficiency in the knowledge and practice of VT among Chinese cardiologists. Specific knowledge and practice support requirements were identified, which provide a foundation for further development and optimization of a CDSS. Moreover, it is important to consider clinicians' specialization levels and years of practice for effective and personalized support.

Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information?

This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.

Data Representation Structure to Support Clinical Decision-Making in the Pediatric Intensive Care Unit: Interview Study and Preliminary Decision Support Interface Design

BACKGROUND

Clinical decision-making is a complex cognitive process that relies on the interpretation of a large variety of data from different sources and involves the use of knowledge bases and scientific recommendations. The representation of clinical data plays a key role in the speed and efficiency of its interpretation. In addition, the increasing use of clinical decision support systems (CDSSs) provides assistance to clinicians in their practice, allowing them to improve patient outcomes. In the pediatric intensive care unit (PICU), clinicians must process high volumes of data and deal with ever-growing workloads. As they use multiple systems daily to assess patients' status and to adjust the health care plan, including electronic health records (EHR), clinical systems (eg, laboratory, imaging and pharmacy), and connected devices (eg, bedside monitors, mechanical ventilators, intravenous pumps, and syringes), clinicians rely mostly on their judgment and ability to trace relevant data for decision-making. In these circumstances, the lack of optimal data structure and adapted visual representation hinder clinician's cognitive processes and clinical decision-making skills.

OBJECTIVE

In this study, we designed a prototype to optimize the representation of clinical data collected from existing sources (eg, EHR, clinical systems, and devices) via a structure that supports the integration of a home-developed CDSS in the PICU. This study was based on analyzing end user needs and their clinical workflow.

METHODS

First, we observed clinical activities in a PICU to secure a better understanding of the workflow in terms of staff tasks and their use of EHR on a typical work shift. Second, we conducted interviews with 11 clinicians from different staff categories (eg, intensivists, fellows, nurses, and nurse practitioners) to compile their needs for decision support. Third, we structured the data to design a prototype that illustrates the proposed representation. We used a brain injury care scenario to validate the relevance of integrated data and the utility of main functionalities in a clinical context. Fourth, we held design meetings with 5 clinicians to present, revise, and adapt the prototype to meet their needs.

RESULTS

We created a structure with 3 levels of abstraction-unit level, patient level, and system level-to optimize clinical data representation and display for efficient patient assessment and to provide a flexible platform to host the internally developed CDSS. Subsequently, we designed a preliminary prototype based on this structure.

CONCLUSIONS

The data representation structure allows prioritizing patients via criticality indicators, assessing their conditions using a personalized dashboard, and monitoring their courses based on the evolution of clinical values. Further research is required to define and model the concepts of criticality, problem recognition, and evolution. Furthermore, feasibility tests will be conducted to ensure user satisfaction.

Detecting atypical alert behavior through statistical process control: Clinical decision support alert frequency visualizations

Clinical decision support (CDS) alerts are designed to work according to a set of clearly defined criteria and have the potential to improve clinical care. To efficiently and proactively review abnormally functioning CDS alerts, we postulate that the introduction of a dashboard with statistical process control (SPC) charting will lead to effective detection of erratic alert behavior. We identified custom CDS alerts from an academic medical center that were recorded and monitored in a longitudinal fashion and the data warehouses where this information was stored. We created a dashboard of alert frequency using SPC charts, applied SPC rules for classification of variation, and validated dashboard data. From June-August 2022, the dashboard effectively pulled in data to visually depict alert behavior. SPC-defined parameters for standard deviation from the mean were applied to visualizations and allowed for rapid review of alerts with greatest variation. These alerts were subsequently investigated, and it was determined that they were functioning correctly. The most profound abnormalities detected during implementation reflected changes in practice and not system errors, though further investigation into thresholds for statistical significance will benefit this field. We conclude that SPC visualizations are a time-efficient and effective method of identifying CDS malfunctions.

Multidisciplinary Development and Initial Validation of a Clinical Knowledge Base on Chronic Respiratory Diseases for mHealth Decision Support Systems

Most mobile health (mHealth) decision support systems currently available for chronic obstructive respiratory diseases (CORDs) are not supported by clinical evidence or lack clinical validation. The development of the knowledge base that will feed the clinical decision support system is a crucial step that involves the collection and systematization of clinical knowledge from relevant scientific sources and its representation in a human-understandable and computer-interpretable way. This work describes the development and initial validation of a clinical knowledge base that can be integrated into mHealth decision support systems developed for patients with CORDs. A multidisciplinary team of health care professionals with clinical experience in respiratory diseases, together with data science and IT professionals, defined a new framework that can be used in other evidence-based systems. The knowledge base development began with a thorough review of the relevant scientific sources (eg, disease guidelines) to identify the recommendations to be implemented in the decision support system based on a consensus process. Recommendations were selected according to predefined inclusion criteria: (1) applicable to individuals with CORDs or to prevent CORDs, (2) directed toward patient self-management, (3) targeting adults, and (4) within the scope of the knowledge domains and subdomains defined. Then, the selected recommendations were prioritized according to (1) a harmonized level of evidence (reconciled from different sources); (2) the scope of the source document (international was preferred); (3) the entity that issued the source document; (4) the operability of the recommendation; and (5) health care professionals' perceptions of the relevance, potential impact, and reach of the recommendation. A total of 358 recommendations were selected. Next, the variables required to trigger those recommendations were defined (n=116) and operationalized into logical rules using Boolean logical operators (n=405). Finally, the knowledge base was implemented in an intelligent individualized coaching component and pretested with an asthma use case. Initial validation of the knowledge base was conducted internally using data from a population-based observational study of individuals with or without asthma or rhinitis. External validation of the appropriateness of the recommendations with the highest priority level was conducted independently by 4 physicians. In addition, a strategy for knowledge base updates, including an easy-to-use rules editor, was defined. Using this process, based on consensus and iterative improvement, we developed and conducted preliminary validation of a clinical knowledge base for CORDs that translates disease guidelines into personalized patient recommendations. The knowledge base can be used as part of mHealth decision support systems. This process could be replicated in other clinical areas.

Evaluation of clinical decision support systems in oncology: An updated systematic review

With increasing reliance on technology in oncology, the impact of digital clinical decision support (CDS) tools needs to be examined. A systematic review update was conducted and peer-reviewed literature from 2016 to 2022 were included if CDS tools were used for live decision making and comparatively assessed quantitative outcomes. 3369 studies were screened and 19 were included in this updated review. Combined with a previous review of 24 studies, a total of 43 studies were analyzed. Improvements in outcomes were observed in 42 studies, and 34 of these were of statistical significance. Computerized physician order entry and clinical practice guideline systems comprise the greatest number of evaluated CDS tools (13 and 10 respectively), followed by those that utilize patient-reported outcomes (8), clinical pathway systems (8) and prescriber alerts for best-practice advisories (4). Our review indicates that CDS can improve guideline adherence, patient-centered care, and care delivery processes in oncology.

Impact of a passive clinical decision support tool on potentially inappropriate medications (PIM) use in older adult patients

BACKGROUND

Medication related clinical decision support (CDS) interventions may improve patient safety. In older patient populations, there has been effort placed in reducing exposure to potentially inappropriate medications (PIMs). After years of reducing exposure of older adults in our hospitals to PIMs through multi-component interventions, our system chose to expand the scope and attempt a new strategy to lessen alert burden for providers and pharmacists. Based on the American Geriatric Society Beers Criteria and internal data, a passive CDS approach, termed "geriatric context" was established to recommend appropriate medication selection including lower dosage amounts and frequency of administration in older adults.

METHODS

Retrospective descriptive study examining change in a pre and post implementation analysis of medication usage patterns between two 9-month time periods in 2019 and 2021 in patients age ≥65 years across an 8-hospital health system. The primary endpoint is the percentage of each medication intervened with an ordered dose and frequency outside of alignment with recommended context parameters. Secondary endpoints include total daily dose (TDD) and average dose (AD) per patient of the individual PIMs. Exploratory endpoints include frequency of active alerts fired by the CPOE and overridden by providers.

RESULTS

A total of 62,738 older adult hospital admissions are included in the overall study period, with 32,969 pre-implementation and 29,769 post-implementation. Haloperidol showed the greatest reduction in inappropriate doses from 41.5% to 21.4% (p < 0.001) of orders, followed by reduction in inappropriate frequencies in orders for diphenhydramine from 57.2% to 39.7% (p < 0.001). Secondary endpoints showed favorable reductions across 11 of the 16 medications in both TDD and AD administered. Exploratory analysis with select medications showed reductions in frequency of alerts fired and overridden.

CONCLUSIONS

Utilization of a passive CDS positively influences prescribing patterns for older adults and reduces the alert burden to ordering providers.

Examining primary care provider experiences with using a clinical decision support tool for pain management

Objective

To evaluate primary care provider (PCP) experiences using a clinical decision support (CDS) tool over 16 months following a user-centered design process and implementation.

Materials and Methods

We conducted a qualitative evaluation of the Chronic Pain OneSheet (OneSheet), a chronic pain CDS tool. OneSheet provides pain- and opioid-related risks, benefits, and treatment information for patients with chronic pain to PCPs. Using the 5 Rights of CDS framework, we conducted and analyzed semi-structured interviews with 19 PCPs across 2 academic health systems.

Results

PCPs stated that OneSheet mostly contained the right information required to treat patients with chronic pain and was correctly located in the electronic health record. PCPs used OneSheet for distinct subgroups of patients with chronic pain, including patients prescribed opioids, with poorly controlled pain, or new to a provider or clinic. PCPs reported variable workflow integration and selective use of certain OneSheet features driven by their preferences and patient population. PCPs recommended broadening OneSheet access to clinical staff and patients for data entry to address clinician time constraints.

Discussion

Differences in patient subpopulations and workflow preferences had an outsized effect on CDS tool use even when the CDS contained the right information identified in a user-centered design process.

Conclusions

To increase adoption and use, CDS design and implementation processes may benefit from increased tailoring that accommodates variation and dynamics among patients, visits, and providers.

Impact of model-informed precision dosing on achievement of vancomycin exposure targets in pediatric patients with cystic fibrosis

BACKGROUND

Vancomycin is commonly used to treat acute pulmonary exacerbations in pediatric patients with cystic fibrosis (CF) and a history of methicillin-resistant Staphylococcus aureus. Optimizing vancomycin exposure during therapy is essential and area under-the-curve (AUC)-guided dosing is now recommended. Model-informed precision dosing (MIPD) utilizing Bayesian forecasting is a powerful approach that can support AUC-guided dose individualization. The objective of the current study was to examine the impact of implementing an AUC-guided dose individualization approach supported via a MIPD clinical decision support (CDS) tool on vancomycin exposure, target attainment rate, and safety in pediatric patients with CF treated with vancomycin during clinical care.

METHODS

A retrospective chart review was performed in patients with CF at a single children's hospital comparing pre- and post-implementation of a MIPD approach for vancomycin supported by a cloud-based, CDS tool integrated into the electronic health record (EHR). In the pre-MIPD period, vancomycin starting doses of 60 mg/kg/day (<13 years) or 45 mg/kg/day (≥13 years) were used. Dose adjustment was guided by therapeutic drug monitoring (TDM) with a target trough 10-20 mg/L. In the post-MIPD period, starting dose and dose adjustment were based on the MIPD CDS tool predictions with a target 24 h AUC (AUC24 ) 400-600 mg*h/L. Exposure and target achievement rates were retrospectively calculated and compared. Rates of acute kidney injury (AKI) were also compared.

RESULTS

Overall, 23 patient courses were included in the pre-MIPD period and 21 patient courses in the post-MIPD period. In the post-MIPD period, an individualized MIPD starting dose resulted in 71% of patients achieving target AUC24 compared to 39% in the pre-MIPD period (p < 0.05). After the first TDM and dose adjustment, target AUC24 achievement was also higher post-MIPD versus pre-MIPD (86% vs. 57%; p < 0.05). AKI rates were low and similar between periods (pre-MIPD 8.7% vs. post-MIPD 9.5%; p = 0.9).

CONCLUSION

An MIPD approach implemented within a cloud-based, EHR-integrated CDS tool safely supported vancomycin AUC-guided dosing and resulted in high rates of target achievement.

The influence of explainable vs non-explainable clinical decision support systems on rapid triage decisions: a mixed methods study

BACKGROUND

During the COVID-19 pandemic, a variety of clinical decision support systems (CDSS) were developed to aid patient triage. However, research focusing on the interaction between decision support systems and human experts is lacking.

METHODS

Thirty-two physicians were recruited to rate the survival probability of 59 critically ill patients by means of chart review. Subsequently, one of two artificial intelligence systems advised the physician of a computed survival probability. However, only one of these systems explained the reasons behind its decision-making. In the third step, physicians reviewed the chart once again to determine the final survival probability rating. We hypothesized that an explaining system would exhibit a higher impact on the physicians' second rating (i.e., higher weight-on-advice).

RESULTS

The survival probability rating given by the physician after receiving advice from the clinical decision support system was a median of 4 percentage points closer to the advice than the initial rating. Weight-on-advice was not significantly different (p = 0.115) between the two systems (with vs without explanation for its decision). Additionally, weight-on-advice showed no difference according to time of day or between board-qualified and not yet board-qualified physicians. Self-reported post-experiment overall trust was awarded a median of 4 out of 10 points. When asked after the conclusion of the experiment, overall trust was 5.5/10 (non-explaining median 4 (IQR 3.5-5.5), explaining median 7 (IQR 5.5-7.5), p = 0.007).

CONCLUSIONS

Although overall trust in the models was low, the median (IQR) weight-on-advice was high (0.33 (0.0-0.56)) and in line with published literature on expert advice. In contrast to the hypothesis, weight-on-advice was comparable between the explaining and non-explaining systems. In 30% of cases, weight-on-advice was 0, meaning the physician did not change their rating. The median of the remaining weight-on-advice values was 50%, suggesting that physicians either dismissed the recommendation or employed a "meeting halfway" approach. Newer technologies, such as clinical reasoning systems, may be able to augment the decision process rather than simply presenting unexplained bias.

Comparative effectiveness of generic commercial versus locally customized clinical decision support tools to reduce prescription of nonsteroidal anti-inflammatory drugs for patients with heart failure

OBJECTIVE

To compare the effectiveness of 2 clinical decision support (CDS) tools to avoid prescription of nonsteroidal anti-inflammatory drugs (NSAIDs) in patients with heart failure (HF): a "commercial" and a locally "customized" alert.

METHODS

We conducted a retrospective cohort study of 2 CDS tools implemented within a large integrated health system. The commercial CDS tool was designed according to third-party drug content and EHR vendor specifications. The customized CDS tool underwent a user-centered design process informed by implementation science principles, with input from a cross disciplinary team. The customized CDS tool replaced the commercial CDS tool. Data were collected from the electronic health record via analytic reports and manual chart review. The primary outcome was effectiveness, defined as whether the clinician changed their behavior and did not prescribe an NSAID.

RESULTS

A random sample of 366 alerts (183 per CDS tool) was evaluated that represented 355 unique patients. The commercial CDS tool was effective for 7 of 172 (4%) patients, while the customized CDS tool was effective for 81 of 183 (44%) patients. After adjusting for age, chronic kidney disease, ejection fraction, NYHA class, concurrent prescription of an opioid or acetaminophen, visit type (inpatient or outpatient), and clinician specialty, the customized alerts were at 24.3 times greater odds of effectiveness compared to the commercial alerts (OR: 24.3 CI: 10.20-58.06).

CONCLUSION

Investing additional resources to customize a CDS tool resulted in a CDS tool that was more effective at reducing the total number of NSAID orders placed for patients with HF compared to a commercially available CDS tool.

The effects of computerised decision support systems on nursing and allied health professional performance and patient outcomes: a systematic review and user contextualisation

Background

Computerised decision support systems (CDSS) are widely used by nurses and allied health professionals but their effect on clinical performance and patient outcomes is uncertain.

Objectives

Evaluate the effects of clinical decision support systems use on nurses', midwives' and allied health professionals' performance and patient outcomes and sense-check the results with developers and users.

Eligibility criteria

Comparative studies (randomised controlled trials (RCTs), non-randomised trials, controlled before-and-after (CBA) studies, interrupted time series (ITS) and repeated measures studies comparing) of CDSS versus usual care from nurses, midwives or other allied health professionals.

Information sources

Nineteen bibliographic databases searched October 2019 and February 2021.

Risk of bias

Assessed using structured risk of bias guidelines; almost all included studies were at high risk of bias.

Synthesis of results

Heterogeneity between interventions and outcomes necessitated narrative synthesis and grouping by: similarity in focus or CDSS-type, targeted health professionals, patient group, outcomes reported and study design.

Included studies

Of 36,106 initial records, 262 studies were assessed for eligibility, with 35 included: 28 RCTs (80%), 3 CBA studies (8.6%), 3 ITS (8.6%) and 1 non-randomised trial, a total of 1318 health professionals and 67,595 patient participants. Few studies were multi-site and most focused on decision-making by nurses (71%) or paramedics (5.7%). Standalone, computer-based CDSS featured in 88.7% of the studies; only 8.6% of the studies involved 'smart' mobile or handheld technology. Care processes - including adherence to guidance - were positively influenced in 47% of the measures adopted. For example, nurses' adherence to hand disinfection guidance, insulin dosing, on-time blood sampling, and documenting care were improved if they used CDSS. Patient care outcomes were statistically - if not always clinically - significantly improved in 40.7% of indicators. For example, lower numbers of falls and pressure ulcers, better glycaemic control, screening of malnutrition and obesity, and accurate triaging were features of professionals using CDSS compared to those who were not.

Evidence limitations

Allied health professionals (AHPs) were underrepresented compared to nurses; systems, studies and outcomes were heterogeneous, preventing statistical aggregation; very wide confidence intervals around effects meant clinical significance was questionable; decision and implementation theory that would have helped interpret effects - including null effects - was largely absent; economic data were scant and diverse, preventing estimation of overall cost-effectiveness.

Interpretation

CDSS can positively influence selected aspects of nurses', midwives' and AHPs' performance and care outcomes. Comparative research is generally of low quality and outcomes wide ranging and heterogeneous. After more than a decade of synthesised research into CDSS in healthcare professions other than medicine, the effect on processes and outcomes remains uncertain. Higher-quality, theoretically informed, evaluative research that addresses the economics of CDSS development and implementation is still required.

Future work

Developing nursing CDSS and primary research evaluation.

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in Health and Social Care Delivery Research; 2023. See the NIHR Journals Library website for further project information.

Registration

PROSPERO [number: CRD42019147773].

Development of a machine learning-based clinical decision support system to predict clinical deterioration in patients visiting the emergency department

This study aimed to develop a machine learning-based clinical decision support system for emergency departments based on the decision-making framework of physicians. We extracted 27 fixed and 93 observation features using data on vital signs, mental status, laboratory results, and electrocardiograms during emergency department stay. Outcomes included intubation, admission to the intensive care unit, inotrope or vasopressor administration, and in-hospital cardiac arrest. eXtreme gradient boosting algorithm was used to learn and predict each outcome. Specificity, sensitivity, precision, F1 score, area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve were assessed. We analyzed 303,345 patients with 4,787,121 input data, resampled into 24,148,958 1 h-units. The models displayed a discriminative ability to predict outcomes (AUROC > 0.9), and the model with lagging 6 and leading 0 displayed the highest value. The AUROC curve of in-hospital cardiac arrest had the smallest change, with increased lagging for all outcomes. With inotropic use, intubation, and intensive care unit admission, the range of AUROC curve change with the leading 6 was the highest according to different amounts of previous information (lagging). In this study, a human-centered approach to emulate the clinical decision-making process of emergency physicians has been adopted to enhance the use of the system. Machine learning-based clinical decision support systems customized according to clinical situations can help improve the quality of care.

Determinants of implementing artificial intelligence-based clinical decision support tools in healthcare: a scoping review protocol

INTRODUCTION

Artificial intelligence (AI), the simulation of human intelligence processes by machines, is being increasingly leveraged to facilitate clinical decision-making. AI-based clinical decision support (CDS) tools can improve the quality of care and appropriate use of healthcare resources, and decrease healthcare provider burnout. Understanding the determinants of implementing AI-based CDS tools in healthcare delivery is vital to reap the benefits of these tools. The objective of this scoping review is to map and synthesise determinants (barriers and facilitators) to implementing AI-based CDS tools in healthcare.

METHODS AND ANALYSIS

This scoping review will follow the Joanna Briggs Institute methodology and the Preferred Reporting Items for Systematic reviews and Meta-Analysis extension for Scoping Reviews checklist. The search terms will be tailored to each database, which includes MEDLINE, Embase, CINAHL, APA PsycINFO and the Cochrane Library. Grey literature and references of included studies will also be searched. The search will include studies published from database inception until 10 May 2022. We will not limit searches by study design or language. Studies that either report determinants or describe the implementation of AI-based CDS tools in clinical practice or/and healthcare settings will be included. The identified determinants (barriers and facilitators) will be described by synthesising the themes using the Theoretical Domains Framework. The outcome variables measured will be mapped and the measures of effectiveness will be summarised using descriptive statistics.

ETHICS AND DISSEMINATION

Ethics approval is not required because all data for this study have been previously published. The findings of this review will be published in a peer-reviewed journal and presented at academic conferences. Importantly, the findings of this scoping review will be widely presented to decision-makers, health system administrators, healthcare providers, and patients and family/caregivers as part of an implementation study of an AI-based CDS for the treatment of coronary artery disease.

MyLynch: A Patient-Facing Clinical Decision Support Tool for Genetically-Guided Personalized Medicine in Lynch Syndrome

Lynch syndrome (LS) is a hereditary cancer susceptibility condition associated with varying cancer risks depending on which of the five causative genes harbors a pathogenic variant; however, lifestyle and medical interventions provide options to lower those risks. We developed MyLynch, a patient-facing clinical decision support (CDS) web application that applies genetically-guided personalized medicine (GPM) for individuals with LS. The tool was developed in R Shiny through a patient-focused iterative design process. The knowledge base used to estimate patient-specific risk leveraged a rigorously curated literature review. MyLynch informs LS patients of their personal cancer risks, educates patients on relevant interventions, and provides patients with adjusted risk estimates, depending on the interventions they choose to pursue. MyLynch can improve risk communication between patients and providers while also encouraging communication among relatives with the goal of increasing cascade testing. As genetic panel testing becomes more widely available, GPM will play an increasingly important role in patient care, and CDS tools offer patients and providers tailored information to inform decision-making. MyLynch provides personalized cancer risk estimates and interventions to lower these risks for patients with LS.

Barriers and Facilitators to the Use of Clinical Decision Support Systems in Primary Care: A Mixed-Methods Systematic Review

PURPOSE

To identify and quantify the barriers and facilitators to the use of clinical decision support systems (CDSSs) by primary care professionals (PCPs).

METHODS

A mixed-methods systematic review was conducted using a sequential synthesis design. PubMed/MEDLINE, PsycInfo, Embase, CINAHL, and the Cochrane library were searched in July 2021. Studies that evaluated CDSSs providing recommendations to PCPs and intended for use during a consultation were included. We excluded CDSSs used only by patients, described as concepts or prototypes, used with simulated cases, and decision supports not considered as CDSSs. A framework synthesis was performed according to the HOT-fit framework (Human, Organizational, Technology, Net Benefits), then a quantitative synthesis evaluated the impact of the HOT-fit categories on CDSS use.

RESULTS

A total of 48 studies evaluating 45 CDSSs were included, and 186 main barriers or facilitators were identified. Qualitatively, barriers and facilitators were classified as human (eg, perceived usefulness), organizational (eg, disruption of usual workflow), and technological (eg, CDSS user-friendliness), with explanatory elements. The greatest barrier to using CDSSs was an increased workload. Quantitatively, the human and organizational factors had negative impacts on CDSS use, whereas the technological factor had a neutral impact and the net benefits dimension a positive impact.

CONCLUSIONS

Our findings emphasize the need for CDSS developers to better address human and organizational issues, in addition to technological challenges. We inferred core CDSS features covering these 3 factors, expected to improve their usability in primary care.

Combining text mining with clinical decision support in clinical practice: a scoping review

OBJECTIVE

Combining text mining (TM) and clinical decision support (CDS) could improve diagnostic and therapeutic processes in clinical practice. This review summarizes current knowledge of the TM-CDS combination in clinical practice, including their intended purpose, implementation in clinical practice, and barriers to such implementation.

MATERIALS AND METHODS

A search was conducted in PubMed, EMBASE, and Cochrane Library databases to identify full-text English language studies published before January 2022 with TM-CDS combination in clinical practice.

RESULTS

Of 714 identified and screened unique publications, 39 were included. The majority of the included studies are related to diagnosis (n = 26) or prognosis (n = 11) and used a method that was developed for a specific clinical domain, document type, or application. Most of the studies selected text containing parts of the electronic health record (EHR), such as reports (41%, n = 16) and free-text narratives (36%, n = 14), and 23 studies utilized a tool that had software "developed for the study". In 15 studies, the software source was openly available. In 79% of studies, the tool was not implemented in clinical practice. Barriers to implement these tools included the complexity of natural language, EHR incompleteness, validation and performance of the tool, lack of input from an expert team, and the adoption rate among professionals.

DISCUSSION/CONCLUSIONS

The available evidence indicates that the TM-CDS combination may improve diagnostic and therapeutic processes, contributing to increased patient safety. However, further research is needed to identify barriers to implementation and the impact of such tools in clinical practice.

Clinical decision support system for hypertension medication based on knowledge graph

BACKGROUND

High prevalence of hypertension and complicated medication knowledge have presented challenges to hypertension clinicians and general practitioners. Clinical decision support systems (CDSSs) are developed to aid clinicians in decision making. Current clinical knowledge is stored in fixed templates, which are not intuitive for clinicians and limit the knowledge reusability. Knowledge graphs (KGs) store knowledge in a way that is not only intuitive to humans but also processable by computers directly. However, existing medical KGs such as UMLS and CMeKG are general purpose and thus lack enough knowledge to enable hypertension medication.

METHODS

We first construct a KG specific to hypertension medication according to the Chinese hypertension guideline and then develop the corresponding CDSS to implement hypertension medication and knowledge management. Current advances in knowledge graph representation and modelling are researched and applied in the complex medical knowledge representation. Traditional knowledge representation and KG representation are innovatively combined in the storage of the KG to enable convenient knowledge management and easy application by the CDSS. Along a predefined reasoning path in the KG, the CDSS finally accomplishes the hypertension medication by applying knowledge stored in the KG. 124 health records of a hypertension Chief Physician from Beijing Anzhen Hospital, Capital Medical University, are collected to evaluate the system metrics on the single drug recommendation task.

RESULTS AND CONCLUSION

The proposed CDSS has functions of medication knowledge graph management and hypertension medication decision support. With elaborate design on knowledge representation, knowledge management is intuitive and convenient. By virtue of the KG, medication recommendations are highly visualized and explainable. Experiments on 124 health records with 90% guideline compliance collected from hospitals in single class recommendation task achieve 91%, 83% and 77% on recall, hit@3 and MRR metrics respectively, which demonstrates the quality of the KG and effectiveness of the system.

A data-driven approach to clinical decision support in tinnitus retraining therapy

Background

Tinnitus, known as “ringing in the ears”, is a widespread and frequently disabling hearing disorder. No pharmacological treatment exists, but clinical management techniques, such as tinnitus retraining therapy (TRT), prove effective in helping patients. Although effective, TRT is not widely offered, due to scarcity of expertise and complexity because of a high level of personalization. Within this study, a data-driven clinical decision support tool is proposed to guide clinicians in the delivery of TRT.

Methods

This research proposes the formulation of data analytics models, based on supervised machine learning (ML) techniques, such as classification models and decision rules for diagnosis, and action rules for treatment to support the delivery of TRT. A knowledge-based framework for clinical decision support system (CDSS) is proposed as a UI-based Java application with embedded WEKA predictive models and Java Expert System Shell (JESS) rule engine with a pattern-matching algorithm for inference (Rete). The knowledge base is evaluated by the accuracy, coverage, and explainability of diagnostics predictions and treatment recommendations.

Results

The ML methods were applied to a clinical dataset of tinnitus patients from the Tinnitus and Hyperacusis Center at Emory University School of Medicine, which describes 555 patients and 3,000 visits. The validated ML classification models for diagnosis and rules: association and actionable treatment patterns were embedded into the knowledge base of CDSS. The CDSS prototype was tested for accuracy and explainability of the decision support, with preliminary testing resulting in an average of 80% accuracy, satisfactory coverage, and explainability.

Conclusions

The outcome is a validated prototype CDS system that is expected to facilitate the TRT practice.

Explainability in medicine in an era of AI-based clinical decision support systems

The combination of “Big Data” and Artificial Intelligence (AI) is frequently promoted as having the potential to deliver valuable health benefits when applied to medical decision-making. However, the responsible adoption of AI-based clinical decision support systems faces several challenges at both the individual and societal level. One of the features that has given rise to particular concern is the issue of explainability, since, if the way an algorithm arrived at a particular output is not known (or knowable) to a physician, this may lead to multiple challenges, including an inability to evaluate the merits of the output. This “opacity” problem has led to questions about whether physicians are justified in relying on the algorithmic output, with some scholars insisting on the centrality of explainability, while others see no reason to require of AI that which is not required of physicians. We consider that there is merit in both views but find that greater nuance is necessary in order to elucidate the underlying function of explainability in clinical practice and, therefore, its relevance in the context of AI for clinical use. In this paper, we explore explainability by examining what it requires in clinical medicine and draw a distinction between the function of explainability for the current patient versus the future patient. This distinction has implications for what explainability requires in the short and long term. We highlight the role of transparency in explainability, and identify semantic transparency as fundamental to the issue of explainability itself. We argue that, in day-to-day clinical practice, accuracy is sufficient as an “epistemic warrant” for clinical decision-making, and that the most compelling reason for requiring explainability in the sense of scientific or causal explanation is the potential for improving future care by building a more robust model of the world. We identify the goal of clinical decision-making as being to deliver the best possible outcome as often as possible, and find—that accuracy is sufficient justification for intervention for today’s patient, as long as efforts to uncover scientific explanations continue to improve healthcare for future patients.

Implementing evidence-based anticoagulant prescribing: User-centered design findings and recommendations

Background

Direct oral anticoagulants (DOACs) are widely used medications with an unacceptably high rate of prescription errors and are a leading cause of adverse drug events. Clinical decision support, including medication alerts, can be an effective implementation strategy to reduce prescription errors, but quality is often inconsistent. User‐centered design (UCD) approaches can improve the effectiveness of alerts.

Objectives

To design effective DOAC prescription alerts through UCD and develop a set of generalizable design recommendations

Methods

This study used an iterative UCD process with practicing clinicians. In three rapid iterative design and assessment stages, prototype alert designs were created and refined using a test electronic health record (EHR) environment and simulated patients. We identified key emergent themes across all user observations and interviews. The themes and final designs were used to derive a set of design guidelines.

Results

Our UCD sample comprised 13 prescribers, including advanced practice providers, physicians in training, primary care physicians, and cardiologists. The resulting alert designs embody our design recommendations, which include establishing intended indication, clarifying dosing by renal function, tailoring alert language in drug interactions, facilitating trust in alerts, and minimizing interaction overhead.

Conclusions

Through a robust UCD process, we have identified key recommendations for implementing medication alerts aimed at improving evidence‐based DOAC prescribing. These recommendations may be applicable to the implementation of DOAC alerts in any EHR systems.

Implementation of an Electronic Health Record Integrated Clinical Pathway Improves Adherence to COVID-19 Hospital Care Guidelines

BACKGROUND

During the COVID-19 pandemic, frequently changing guidelines presented challenges to emergency department (ED) clinicians. The authors implemented an electronic health record (EHR)-integrated clinical pathway that could be accessed by clinicians within existing workflows when caring for patients under investigation (PUI) for COVID-19. The objective was to examine the association between clinical pathway utilization and adherence to institutional best practice treatment recommendations for COVID-19.

METHODS

The authors conducted an observational analysis of all ED patients seen in a health system inclusive of seven EDs between March 18, 2020, and April 20, 2021. They implemented the pathway as an interactive flow chart that allowed clinicians to place orders while viewing the most up-to-date institutional guidance. Primary outcomes were proportion of admitted PUIs receiving dexamethasone and aspirin in the ED, and secondary outcome was time to delivering treatment.

RESULTS

A total of 13 269 patients were admitted PUIs. The pathway was used by 40.6% of ED clinicians. When clinicians used the pathway, patients were more likely to be prescribed aspirin (OR, 7.15; 95% CI, 6.2-8.26) and dexamethasone (10.4; 8.85-12.2). For secondary outcomes, clinicians using the pathway had statistically significant ( P < 0.0001) improvement in timeliness of ordering medications and admission to the hospital. Aspirin, dexamethasone, and admission order time were improved by 103.89, 94.34, and 121.94 minutes, respectively.

CONCLUSIONS

The use of an EHR-integrated clinical pathway improved clinician adherence to changing COVID-19 treatment guidelines and timeliness to associated medication administration. As pathways continue to be implemented, their effects on improving patient outcomes and decreasing disparities in patient care should be further examined.

Usability, acceptability, and implementation strategies for the Exercise in Cancer Evaluation and Decision Support (EXCEEDS) algorithm: a Delphi study

INTRODUCTION

Oncology guidelines recommend participation in cancer rehabilitation or exercise services (CR/ES) to optimize survivorship. Yet, connecting the right survivor, with the right CR/ES, at the right time remains a challenge. The Exercise in Cancer Evaluation and Decision Support (EXCEEDS) algorithm was developed to enhance CR/ES clinical decision-making and facilitate access to CR/ES. We used Delphi methodology to evaluate usability, acceptability, and determine pragmatic implementation priorities.

METHODS

Participants completed three online questionnaires including (1) simulated case vignettes, (2) 4-item acceptability questionnaire (0-5 pts), and (3) series of items to rank algorithm implementation priorities (potential users, platforms, strategies). To evaluate usability, we used Chi-squared test to compare frequency of accurate pre-exercise medical clearance and CR/ES triage recommendations for case vignettes when using EXCEEDS vs. without. We calculated mean acceptability and inter-rater agreement overall and in 4 domains. We used the Eisenhower Prioritization Method to evaluate implementation priorities.

RESULTS

Participants (N = 133) mostly represented the fields of rehabilitation (69%), oncology (25%), or exercise science (17%). When using EXCEEDS (vs. without), their recommendations were more likely to be guideline concordant for medical clearance (83.4% vs. 66.5%, X² = 26.61, p < .0001) and CR/ES triage (60.9% vs. 51.1%, X² = 73.79, p < .0001). Mean acceptability was M = 3.90 ± 0.47; inter-rater agreement was high for 3 of 4 domains. Implementation priorities include 1 potential user group, 2 platform types, and 9 implementation strategies.

CONCLUSION

This study demonstrates the EXCEEDS algorithm can be a pragmatic and acceptable clinical decision support tool for CR/ES recommendations. Future research is needed to evaluate algorithm usability and acceptability in real-world clinical pathways.

Clinical decision support in cardiovascular medicine

Despite considerable progress in tackling cardiovascular disease over the past 50 years, many gaps in the quality of care for cardiovascular disease remain. Multiple missed opportunities have been identified at every step in the prevention and treatment of cardiovascular disease, such as failure to make risk factor modifications, failure to diagnose cardiovascular disease, and failure to use proper evidence based treatments. With the digital transformation of medicine and advances in health information technology, clinical decision support (CDS) tools offer promise to enhance the efficiency and effectiveness of delivery of cardiovascular care. However, to date, the promise of CDS delivering scalable and sustained value for patient care in clinical practice has not been realized. This article reviews the evidence on key emerging questions around the development, implementation, and regulation of CDS with a focus on cardiovascular disease. It first reviews evidence on the effectiveness of CDS on healthcare process and clinical outcomes related to cardiovascular disease and design features associated with CDS effectiveness. It then reviews the barriers encountered during implementation of CDS in cardiovascular care, with a focus on unintended consequences and strategies to promote successful implementation. Finally, it reviews the legal and regulatory environment of CDS with specific examples for cardiovascular disease.

Designing Flexible Longitudinal Regimens: Supporting Clinician Planning for Discontinuation of Psychiatric Drugs

Clinical decision support tools have typically focused on one-time support for diagnosis or prognosis, but have the ability to support providers in longitudinal planning of patient care regimens amidst infrastructural challenges. We explore an opportunity for technology support for discontinuing antidepressants, where clinical guidelines increasingly recommend gradual discontinuation over abruptly stopping to avoid withdrawal symptoms, but providers have varying levels of experience and diverse strategies for supporting patients through discontinuation. We conducted two studies with 12 providers, identifying providers’ needs in developing discontinuation plans and deriving design guidelines. We then iteratively designed and implemented AT Planner, instantiating the guidelines by projecting taper schedules and providing flexibility for adjustment. Provider feedback on AT Planner highlighted that discontinuation plans required balancing interpersonal and infrastructural constraints and surfaced the need for different technological support based on clinical experience. We discuss the benefits and challenges of incorporating flexibility and advice into clinical planning tools.

Clinician Acceptance of Order Sets for Pain Management: A Survey in Two Urban Hospitals

BACKGROUND

 Order sets are a clinical decision support (CDS) tool in computerized provider order entry systems. Order set use has been associated with improved quality of care. Particularly related to opioids and pain management, order sets have been shown to standardize and reduce the prescription of opioids. However, clinician-level barriers often limit the uptake of this CDS modality.

OBJECTIVE

 To identify the barriers to order sets adoption, we surveyed clinicians on their training, knowledge, and perceptions related to order sets for pain management.

METHODS

 We distributed a cross-sectional survey between October 2020 and April 2021 to clinicians eligible to place orders at two campuses of a major academic medical center. Survey questions were adapted from the widely used framework of Unified Theory of Acceptance and Use of Technology. We hypothesize that performance expectancy (PE) and facilitating conditions (FC) are associated with order set use. Survey responses were analyzed using logistic regression.

RESULTS

 The intention to use order sets for pain management was associated with PE to existing order sets, social influence (SI) by leadership and peers, and FC for electronic health record (EHR) training and function integration. Intention to use did not significantly differ by gender or clinician role. Moderate differences were observed in the perception of the effort of, and FC for, order set use across gender and roles of clinicians, particularly emergency medicine and internal medicine departments.

CONCLUSION

 This study attempts to identify barriers to the adoption of order sets for pain management and suggests future directions in designing and implementing CDS systems that can improve order sets adoption by clinicians. Study findings imply the importance of order set effectiveness, peer influence, and EHR integration in determining the acceptability of the order sets.

An Opportunity in Cancer Prevention: Human Papillomavirus Vaccine Delivery in the Hospital

OBJECTIVE

Pediatric hospitalizations are a missed opportunity for delivery of the human papilloma virus (HPV) vaccination. In this study, the authors' aim was to increase HPV vaccination rates among adolescents cared for by the pediatric hospital medicine (PHM) service at our academic children's hospital.

METHODS

This quality improvement (QI) study included adolescents ≥13 years who were discharged from PHM. Interventions included: modification of discharge order sets to include vaccination status and provider training seminars regarding the delivery of the HPV vaccine. Follow-up materials were distributed to providers by e-mail. The primary outcome measure was adolescent HPV vaccination rates. Secondary outcome measures were adolescent meningococcal vaccination rates and accuracy of immunization status documentation. The balancing measure was length of stay (LOS). Data were collected via chart review. Statistical process control charts were used to analyze for special cause variation.

RESULTS

From May 2019 through February 2020, 440 patients were included in this analysis. Throughout the study, HPV and meningococcal vaccination rates increased from a baseline median of 4.6% to 21.2% and 8.3% to 26.6%, respectively. HPV vaccination was not significantly associated with sex, HPV dose due, or admitting service. Accuracy of immunization status documentation and LOS remained unchanged.

CONCLUSIONS

Using QI methodology we were successful in increasing HPV and meningococcal vaccination rates among hospitalized adolescents. Considering the relationship of these 2 vaccines is a potential topic of future work. Discerning the correct immunization status at time of admission may be a potential opportunity for improvement in future work.

PercolationDF: A percolation-based medical diagnosis framework

Goal: With the continuing shortage and unequal distribution of medical resources, our objective is to develop a general diagnosis framework that utilizes a smaller amount of electronic medical records (EMRs) to alleviate the problem that the data volume requirement of prevailing models is too vast for medical institutions to afford. Methods: The framework proposed contains network construction, network expansion, and disease diagnosis methods. In the first two stages above, the knowledge extracted from EMRs is utilized to build and expense an EMR-based medical knowledge network (EMKN) to model and represent the medical knowledge. Then, percolation theory is modified to diagnose EMKN. Result: Facing the lack of data, our framework outperforms naïve Bayes networks, neural networks and logistic regression, especially in the top-10 recall. Out of 207 test cases, 51.7% achieved 100% in the top-10 recall, 21% better than what was achieved in one of our previous studies. Conclusion: The experimental results show that the proposed framework may be useful for medical knowledge representation and diagnosis. The framework effectively alleviates the lack of data volume by inferring the knowledge modeled in EMKN. Significance: The proposed framework not only has applications for diagnosis but also may be extended to other domains to represent and model the knowledge and inference on the representation.

Coordinated use of Health Level 7 standards to support clinical decision support: Case study with shared decision making and drug-drug interactions

BACKGROUND

Despite advances in interoperability standards, it remains challenging and often costly to share clinical decision support (CDS) across healthcare organizations. This is due in part to limited coordination among CDS components. To improve coordination of CDS components, Health Level 7 (HL7) has developed a suite of interoperability standards with Fast Health Interoperability Resources (FHIR) specification as a common information model. Evidence is needed to determine the feasibility of implementing these CDS components; therefore, the objective of this study was to investigate the coordination of emerging HL7 standards with modular CDS architecture components.

METHODS

We used a modular, standards-based architecture consisting of four components: data, logic, services, and applications. The implementation use-case was an application to support shared decision making in the context of drug-drug interactions (DDInteract).

RESULTS

DDInteract uses FHIR as the data representation model, Clinical Quality Language for logic representation, CDS Hooks for the services layer, and Substitutable Medical Apps Reusable Technologies for application integration. DDInteract was first implemented in a sandbox environment and then in an electronic health record (Epic®) test environment. DDInteract can be integrated in clinical workflows through on-demand access from a menu or through CDS Hooks upon opening a patient's record or placing a medication order.

CONCLUSION

In the context of drug interactions, DDInteract is the first application to leverage a full stack of emerging interoperability standards for each component of modular CDS architecture. The demonstrated feasibility of interoperable components can be generalized to other modular CDS applications.

A Data-Driven Algorithm to Recommend Initial Clinical Workup for Outpatient Specialty Referral: Algorithm Development and Validation Using Electronic Health Record Data and Expert Surveys

BACKGROUND

Millions of people have limited access to specialty care. The problem is exacerbated by ineffective specialty visits due to incomplete prereferral workup, leading to delays in diagnosis and treatment. Existing processes to guide prereferral diagnostic workup are labor-intensive (ie, building a consensus guideline between primary care doctors and specialists) and require the availability of the specialists (ie, electronic consultation).

OBJECTIVE

Using pediatric endocrinology as an example, we develop a recommender algorithm to anticipate patients' initial workup needs at the time of specialty referral and compare it to a reference benchmark using the most common workup orders. We also evaluate the clinical appropriateness of the algorithm recommendations.

METHODS

Electronic health record data were extracted from 3424 pediatric patients with new outpatient endocrinology referrals at an academic institution from 2015 to 2020. Using item co-occurrence statistics, we predicted the initial workup orders that would be entered by specialists and assessed the recommender's performance in a holdout data set based on what the specialists actually ordered. We surveyed endocrinologists to assess the clinical appropriateness of the predicted orders and to understand the initial workup process.

RESULTS

Specialists (n=12) indicated that <50% of new patient referrals arrive with complete initial workup for common referral reasons. The algorithm achieved an area under the receiver operating characteristic curve of 0.95 (95% CI 0.95-0.96). Compared to a reference benchmark using the most common orders, precision and recall improved from 37% to 48% (P<.001) and from 27% to 39% (P<.001) for the top 4 recommendations, respectively. The top 4 recommendations generated for common referral conditions (abnormal thyroid studies, obesity, amenorrhea) were considered clinically appropriate the majority of the time by specialists surveyed and practice guidelines reviewed.

CONCLUSIONS

 An item association-based recommender algorithm can predict appropriate specialists' workup orders with high discriminatory accuracy. This could support future clinical decision support tools to increase effectiveness and access to specialty referrals. Our study demonstrates important first steps toward a data-driven paradigm for outpatient specialty consultation with a tier of automated recommendations that proactively enable initial workup that would otherwise be delayed by awaiting an in-person visit.

The potential for leveraging machine learning to filter medication alerts

OBJECTIVE

To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view.

MATERIALS AND METHODS

We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision.

RESULTS

A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001).

CONCLUSIONS

Machine learning potentially enables the intelligent filtering of medication alerts.

Innovation in the pediatric electronic health record to realize a more effective platform

Commercial electronic health records (EHRs) were first developed to automate business processes. As EHRs developed, design principles focused on transferring existing paper-based documentation to comparable electronic forms. In addition, a strong industry focus on adult healthcare settings and quality measures has limited attention and resources for high priority EHR functionality needed for the unique health care of children. The objective of this paper is to provide a review of innovation in the EHR, that includes a variety of established and emerging technologies that may help realize a more effective EHR in child health settings. A more effective EHR would serve as an electronic hub. Existing EHR infrastructure could provide the foundation upon which new technologies and approaches branch and extend, enabling more rapid and customizable innovation to better meet shifting stakeholder and end-user needs. Among many areas for improvement, key goals of innovation could include technology that relieves ambulatory primary care clinician documentation burden, identifies needs, and supports improved care coordination and outcomes, focused on the following key areas: identification of child and family care needs, decision support, documentation, care coordination, and family communication.

Physician Knowledge Base: Clinical Decision Support Systems

With the introduction of electronic medical records (EMRs), it has become possible to accumulate massive amounts of qualitative medical data. As such, EMRs have become increasingly used in clinical decision support systems (CDSSs). While CDSSs aim to reduce medical errors normally occurring in the process of treating patients by physicians, technical maturity and the completeness of CDSSs do not meet standards for medical use yet. As data further accumulates, CDSS algorithms must be continuously updated to allow CDSSs to perform their core functions. Doing so, however, requires extensive time and manpower investments. In current practice, computational systems already perform a wide variety of functions in medical settings to allow medical staff to focus on other tasks. However, no prior research has evaluated the potential effectiveness of future CDSSs nor analyzed possibilities for their further development. In this article, we evaluate CDSS technology with the consideration that medical staff also understand the core functions of such systems.

Save Face or Save Life: Physicians’ Dilemma in Using Clinical Decision Support Systems

Humans think both rationally and heuristically. So do physicians. Clinical decision support systems (CDSSs) provide advice to physicians that could save patients’ lives, but they could also make physicians feel face loss because of submission to machine intelligence, leading to a perplexing dilemma. Thinking rationally, physicians focus on fulfilling their professional duty to save patients and should follow advice from CDSS to improve care quality. Thinking heuristically, they focus on protecting their authoritative image to maintain face and are inclined to avoid embarrassment by resisting CDSS. Through a longitudinal survey and follow-up interviews with a group of Chinese physicians, we find that the dilemma does exist. Moreover, face loss has a stronger effect on CDSS resistance when physicians have high autonomy. When time pressure is high, perceived usefulness more strongly reduces, whereas face loss more strongly increases CDSS resistance, worsening the dilemma. As face is a universal social concern existing in both Eastern and Western cultures, this research generates insights regarding why physicians are slow in adopting information technology innovations.

Artificial Intelligence: A Shifting Paradigm in Cardio-Cerebrovascular Medicine

The future of healthcare is an organic blend of technology, innovation, and human connection. As artificial intelligence (AI) is gradually becoming a go-to technology in healthcare to improve efficiency and outcomes, we must understand our limitations. We should realize that our goal is not only to provide faster and more efficient care, but also to deliver an integrated solution to ensure that the care is fair and not biased to a group of sub-population. In this context, the field of cardio-cerebrovascular diseases, which encompasses a wide range of conditions-from heart failure to stroke-has made some advances to provide assistive tools to care providers. This article aimed to provide an overall thematic review of recent development focusing on various AI applications in cardio-cerebrovascular diseases to identify gaps and potential areas of improvement. If well designed, technological engines have the potential to improve healthcare access and equitability while reducing overall costs, diagnostic errors, and disparity in a system that affects patients and providers and strives for efficiency.

A Behavioral Economics-Electronic Health Record Module to Promote Appropriate Diabetes Management in Older Adults: Protocol for a Pragmatic Cluster Randomized Controlled Trial

Background

The integration of behavioral economics (BE) principles and electronic health records (EHRs) using clinical decision support (CDS) tools is a novel approach to improving health outcomes. Meanwhile, the American Geriatrics Society has created the Choosing Wisely (CW) initiative to promote less aggressive glycemic targets and reduction in pharmacologic therapy in older adults with type 2 diabetes mellitus. To date, few studies have shown the effectiveness of combined BE and EHR approaches for managing chronic conditions, and none have addressed guideline-driven deprescribing specifically in type 2 diabetes. We previously conducted a pilot study aimed at promoting appropriate CW guideline adherence using BE nudges and EHRs embedded within CDS tools at 5 clinics within the New York University Langone Health (NYULH) system. The BE-EHR module intervention was tested for usability, adoption, and early effectiveness. Preliminary results suggested a modest improvement of 5.1% in CW compliance.

Objective

This paper presents the protocol for a study that will investigate the effectiveness of a BE-EHR module intervention that leverages BE nudges with EHR technology and CDS tools to reduce overtreatment of type 2 diabetes in adults aged 76 years and older, per the CW guideline.

Methods

A pragmatic, investigator-blind, cluster randomized controlled trial was designed to evaluate the BE-EHR module. A total of 66 NYULH clinics will be randomized 1:1 to receive for 18 months either (1) a 6-component BE-EHR module intervention + standard care within the NYULH EHR, or (2) standard care only. The intervention will be administered to clinicians during any patient encounter (eg, in person, telemedicine, medication refill, etc). The primary outcome will be patient-level CW compliance. Secondary outcomes will measure the frequency of intervention component firings within the NYULH EHR, and provider utilization and interaction with the BE-EHR module components.

Results

Study recruitment commenced on December 7, 2020, with the activation of all 6 BE-EHR components in the NYULH EHR.

Conclusions

This study will test the effectiveness of a previously developed, iteratively refined, user-tested, and pilot-tested BE-EHR module aimed at providing appropriate diabetes care to elderly adults, compared to usual care via a cluster randomized controlled trial. This innovative research will be the first pragmatic randomized controlled trial to use BE principles embedded within the EHR and delivered using CDS tools to specifically promote CW guideline adherence in type 2 diabetes. The study will also collect valuable information on clinician workflow and interaction with the BE-EHR module, guiding future research in optimizing the timely delivery of BE nudges within CDS tools. This work will address the effectiveness of BE-inspired interventions in diabetes and chronic disease management.

Trial Registration

ClinicalTrials.gov NCT04181307; https://clinicaltrials.gov/ct2/show/NCT04181307

International Registered Report Identifier (IRRID)

DERR1-10.2196/28723

Predicting inpatient pharmacy order interventions using provider action data

Objective

The widespread deployment of electronic health records (EHRs) has introduced new sources of error and inefficiencies to the process of ordering medications in the hospital setting. Existing work identifies orders that require pharmacy intervention by comparing them to a patient’s medical records. In this work, we develop a machine learning model for identifying medication orders requiring intervention using only provider behavior and other contextual features that may reflect these new sources of inefficiencies.

Materials and Methods

Data on providers’ actions in the EHR system and pharmacy orders were collected over a 2-week period in a major metropolitan hospital system. A classification model was then built to identify orders requiring pharmacist intervention. We tune the model to the context in which it would be deployed and evaluate global and local feature importance.

Results

The resultant model had an area under the receiver-operator characteristic curve of 0.91 and an area under the precision-recall curve of 0.44.

Conclusions

Providers’ actions can serve as useful predictors in identifying medication orders that require pharmacy intervention. Careful model tuning for the clinical context in which the model is deployed can help to create an effective tool for improving health outcomes without using sensitive patient data.

Machine learning models for decision support in epilepsy management: A critical review

PURPOSE

There remain major challenges for the clinician in managing patients with epilepsy effectively. Choosing anti-seizure medications (ASMs) is subject to trial and error. About one-third of patients have drug-resistant epilepsy (DRE). Surgery may be considered for selected patients, but time from diagnosis to surgery averages 20 years. We reviewed the potential use of machine learning (ML) predictive models as clinical decision support tools to help address some of these issues.

METHODS

We conducted a comprehensive search of Medline and Embase of studies that investigated the application of ML in epilepsy management in terms of predicting ASM responsiveness, predicting DRE, identifying surgical candidates, and predicting epilepsy surgery outcomes. Original articles addressing these 4 areas published in English between 2000 and 2020 were included.

RESULTS

We identified 24 relevant articles: 6 on ASM responsiveness, 3 on DRE prediction, 2 on identifying surgical candidates, and 13 on predicting surgical outcomes. A variety of potential predictors were used including clinical, neuropsychological, imaging, electroencephalography, and health system claims data. A number of different ML algorithms and approaches were used for prediction, but only one study utilized deep learning methods. Some models show promising performance with areas under the curve above 0.9. However, most were single setting studies (18 of 24) with small sample sizes (median number of patients 55), with the exception of 3 studies that utilized large databases and 3 studies that performed external validation. There was a lack of standardization in reporting model performance. None of the models reviewed have been prospectively evaluated for their clinical benefits.

CONCLUSION

The utility of ML models for clinical decision support in epilepsy management remains to be determined. Future research should be directed toward conducting larger studies with external validation, standardization of reporting, and prospective evaluation of the ML model on patient outcomes.

Implementing an Electronic Clinical Decision Support Tool Into Routine Care: A Qualitative Study of Stakeholders' Perceptions of a Post-Mastectomy Breast Reconstruction Tool

Objective. To explore barriers and facilitators to implementing an evidence-based clinical decision support (CDS) tool (BREASTChoice) about post-mastectomy breast reconstruction into routine care. Materials and Methods. A stakeholder advisory group of cancer survivors, clinicians who discuss and/or perform breast reconstruction in women with cancer, and informatics professionals helped design and review the interview guide. Based on the Consolidated Framework for Implementation Research (CFIR), we conducted qualitative semistructured interviews with key stakeholders (patients, clinicians, informatics professionals) to explore intervention, setting characteristics, and process-level variables that can impact implementation. Interviews were transcribed, coded, and analyzed based on the CFIR framework using both inductive and deductive methods. Results. Fifty-seven potential participants were contacted; 49 (85.9%) were eligible, and 35 (71.4%) were enrolled, continuing until thematic saturation was reached. Participants consisted of 13 patients, 13 clinicians, and 9 informatics professionals. Stakeholders thought that BREASTChoice was useful and provided patients with an evidence-based source of information about post-mastectomy breast reconstruction, including their personalized risks. They felt that BREASTChoice could support shared decision making, improve workflow, and possibly save consultation time, but were uncertain about the best time to deliver BREASTChoice to patients. Some worried about cost, data availability, and security of integrating the tool into an electronic health record. Most acknowledged the importance of showing clinical utility to gain institutional buy-in and encourage routine adoption. Discussion and Conclusion. Stakeholders felt that BREASTChoice could support shared decision making, improve workflow, and reduce consultation time. Addressing key questions such as cost, data integration, and timing of delivering BREASTChoice could build institutional buy-in for CDS implementation. Results can guide future CDS implementation studies.

Leveraging Clinical Decision Support and Integrated Medical-Dental Electronic Health Records to Implementing Precision in Oral Cancer Risk Assessment and Preventive Intervention

INTRODUCTION

Precision medicine is focused on serving the unique needs of individuals. Oral and oropharyngeal cancer risk assessment identifies individual risk factors while providing support to reduce risk. The objective is to examine potential current and future strategies to broadly implement evidence-based oral and oropharyngeal cancer risk assessment and screening in dental practices throughout the United States.

METHODS

Feasible and effective oral cancer risk assessment and risk reduction strategies, ripe for implementation in dental practice, were identified in the published literature.

RESULTS

The Screening, Brief Intervention, Referral for Treatment (SBIRT) model is a feasible approach to assessing individual oral cancer risk and providing risk reducing interventions in the dental setting. HPV is a more recently identified risk factor that dentistry is well positioned to address. Evidence supporting the utilization of specific risk assessment tools and risk reduction strategies is summarized and future opportunities discussed.

DISCUSSION

Current knowledge of risk factors for oral and oropharyngeal cancers support the recommendation for dental providers to routinely assess all patients for risk factors, educate them about their personal level of cancer risk, and recommend actions to reduce relevant risk factors. Individuals ages 9-26 should be asked about their HPV vaccination status, educated about HPV and oropharyngeal cancer and receive a recommendation to get the HPV vaccination.

Using Electronic Health Record Tools to Decrease Antibiotic Exposure in Infant Sepsis Evaluation

BACKGROUND

Our internal infant sepsis evaluation clinical practice guideline recommends infants with negative culture results who are undergoing sepsis evaluation receive antibiotics until culture results are negative for a maximum of 36 hours. The aims of our project were to decrease the percentage of patients who received >30 hours of administered antibiotic doses (recognizing effective concentrations last until hour 36) and increase 36-hour phrase documentation by using clinical decision support tools.

METHODS

We used quality improvement methodology to study infants aged ≤60 days with negative culture results. The outcome measures were the percentage of patients who received >30 hours of administered antibiotic doses, the percentage of history and physical (H&P) notes that included a statement of the anticipated 36-hour antibiotic discontinuation time (36-hour phrase), and length of stay. The process measure was the use of an illness-specific H&P template or an influencer smartphrase. Balancing measures were readmissions for positive culture results. Interventions included education, an illness-specific H&P template, a criteria-based rule to default to this H&P template, and editing influencer smartphrases.

RESULTS

Over 33 months, 311 patients were included. Percentage of patients who received >30 hours of administered antibiotic doses decreased from 75.6% to 62%. Percentage of H&P notes documenting the 36-hour phrase increased from 4.9% to 75.6%. Illness-specific H&P template and influencer smartphrase usage increased to a mean of 51.5%; length of stay did not change. No readmissions for positive culture results were reported.

CONCLUSIONS

Clinical decision support techniques and educational interventions popularized the "36-hour phrase" and were associated with a reduction in the antibiotic exposure in infants with negative culture results hospitalized for sepsis evaluation.