Clinical decision support models and frameworks: Seeking to address research issues underlying implementation successes and failures

Augmenting clinicians' analytical workflow through task-based integration of data visualizations and algorithmic insights: a user-centered design study

OBJECTIVES

To understand healthcare providers' experiences of using GlucoGuide, a mockup tool that integrates visual data analysis with algorithmic insights to support clinicians' use of patientgenerated data from Type 1 diabetes devices.

MATERIALS AND METHODS

This qualitative study was conducted in three phases. In Phase 1, 11 clinicians reviewed data using commercial diabetes platforms in a think-aloud data walkthrough activity followed by semistructured interviews. In Phase 2, GlucoGuide was developed. In Phase 3, the same clinicians reviewed data using GlucoGuide in a think-aloud activity followed by semistructured interviews. Inductive thematic analysis was used to analyze transcripts of Phase 1 and Phase 3 think-aloud activity and interview.

RESULTS

3 high level tasks, 8 sub-tasks, and 4 challenges were identified in Phase 1. In Phase 2, 3 requirements for GlucoGuide were identified. Phase 3 results suggested that clinicians found GlucoGuide easier to use and experienced a lower cognitive burden as compared to the commercial diabetes data reports that were used in Phase 1. Additionally, GlucoGuide addressed the challenges experienced in Phase 1.

DISCUSSION

The study suggests that the knowledge of analytical tasks and task-specific visualization strategies in implementing features of data interfaces can result in tools that lower the perceived burden of engaging with data. Additionally, supporting clinicians in contextualizing algorithmic insights by visual analysis of relevant data can positively influence clinicians' willingness to leverage algorithmic support.

CONCLUSION

Task-aligned tools that combine multiple data-driven approaches, such as visualization strategies and algorithmic insights, can improve clinicians' experience in reviewing device data.

Nurses' perceptions of the design, implementation, and adoption of machine learning clinical decision support: A descriptive qualitative study

INTRODUCTION

The purpose of this study was to explore nurses' perspectives on Machine Learning Clinical Decision Support (ML CDS) design, development, implementation, and adoption.

DESIGN

Qualitative descriptive study.

METHODS

Nurses (n = 17) participated in semi-structured interviews. Data were transcribed, coded, and analyzed using Thematic analysis methods as described by Braun and Clarke.

RESULTS

Four major themes and 14 sub-themes highlight nurses' perspectives on autonomy in decision-making, the influence of prior experience in shaping their preferences for use of novel CDS tools, the need for clarity in why ML CDS is useful in improving practice/outcomes, and their desire to have nursing integrated in design and implementation of these tools.

CONCLUSION

This study provided insights into nurse perceptions regarding the utility and usability of ML CDS as well as the influence of previous experiences with technology and CDS, change management strategies needed at the time of implementation of ML CDS, the importance of nurse-perceived engagement in the development process, nurse information needs at the time of ML CDS deployment, and the perceived impact of ML CDS on nurse decision making autonomy.

CLINICAL RELEVANCE

This study contributes to the body of knowledge about the use of AI and machine learning (ML) in nursing practice. Through generation of insights drawn from nurses' perspectives, these findings can inform successful design and adoption of ML Clinical Decision Support.

Optimising interruptive clinical decision support alerts for antithrombotic duplicate prescribing in hospital

INTRODUCTION

Duplicate prescribing clinical decision support alerts can prevent important prescribing errors but are frequently the cause of much alert fatigue. Stat dose prescriptions are a known reason for overriding these alerts. This study aimed to evaluate the effect of excluding stat dose prescriptions from duplicate prescribing alerts for antithrombotic medicines on alert burden, prescriber adherence, and prescribing.

MATERIALS AND METHODS

A before (January 1st, 2017 to August 31st, 2022) and after (October 5th, 2022 to September 30th, 2023) study was undertaken of antithrombotic duplicate prescribing alerts and prescribing following a change in alert settings. Alert and prescribing data for antithrombotic medicines were joined, processed, and analysed to compare alert rates, adherence, and prescribing. Alert burden was assessed as alerts per 100 prescriptions. Adherence was measured at the point of the alert as whether the prescriber accepted the alert and following the alert as whether a relevant prescription was ceased within an hour. Co-prescribing of antithrombotic stat dose prescriptions was assessed pre- and post-alert reconfiguration.

RESULTS

Reconfiguration of the alerts reduced the alert rate by 29 % (p < 0.001). The proportion of alerts associated with cessation of antithrombotic duplication significantly increased (32.8 % to 44.5 %, p < 0.001). Adherence at the point of the alert increased 1.2 % (4.8 % to 6.0 %, p = 0.012) and 11.5 % (29.4 % to 40.9 %, p < 0.001) within one hour of the alert. When ceased after the alert over 80 % of duplicate prescriptions were ceased within 2 min of overriding. Antithrombotic stat dose co-prescribing was unchanged for 4 out of 5 antithrombotic duplication alert rules.

CONCLUSION

By reconfiguring our antithrombotic duplicate prescribing alerts, we reduced alert burden and increased alert adherence. Many prescribers ceased duplicate prescribing within 2 min of alert override highlighting the importance of incorporating post-alert measures in accurately determining prescriber alert adherence.

Transforming evidence-based clinical guidelines into implementable clinical decision support services: the CAREPATH study for multimorbidity management

Introduction

The CAREPATH Project aims to develop a patient-centered integrated care platform tailored to older adults with multimorbidity, including mild cognitive impairment (MCI) or mild dementia. Our goal is to empower multidisciplinary care teams to craft personalized holistic care plans while adhering to evidence-based guidelines. This necessitates the creation of clear specifications for clinical decision support (CDS) services, consolidating guidance from multiple evidence-based clinical guidelines. Thus, a co-creation approach involving both clinical and technical experts is essential.

Methods

This paper outlines a robust methodology for generating implementable specifications for CDS services to automate clinical guidelines. We have established a co-creation framework to facilitate collaborative exploration of clinical guidelines between clinical experts and software engineers. We have proposed an open, repeatable, and traceable method for translating evidence-based guideline narratives into implementable specifications of CDS services. Our approach, based on international standards such as CDS-Hooks and HL7 FHIR, enhances interoperability and potential adoption of CDS services across diverse healthcare systems.

Results

This methodology has been followed to create implementable specifications for 65 CDS services, automating CAREPATH consensus guideline consolidating guidance from 25 selected evidence-based guidelines. A total of 296 CDS rules have been formally defined, with input parameters defined as clinical concepts bound to FHIR resources and international code systems. Outputs include 346 well-defined CDS Cards, offering clear guidance for care plan activities and goal suggestions. These specifications have led to the implementation of 65 CDS services integrated into the CAREPATH Adaptive Integrated Care Platform.

Discussion

Our methodology offers a systematic, replicable process for generating CDS specifications, ensuring consistency and reliability across implementation. By fostering collaboration between clinical expertise and technical proficiency, we enhance the quality and relevance of generated specifications. Clear traceability enables stakeholders to track the development process and ensure adherence to guideline recommendations.

User-Centered Framework for Implementation of Technology (UFIT): Development of an Integrated Framework for Designing Clinical Decision Support Tools Packaged With Tailored Implementation Strategies

BACKGROUND

Electronic health record-based clinical decision support (CDS) tools can facilitate the adoption of evidence into practice. Yet, the impact of CDS beyond single-site implementation is often limited by dissemination and implementation barriers related to site- and user-specific variation in workflows and behaviors. The translation of evidence-based CDS from initial development to implementation in heterogeneous environments requires a framework that assures careful balancing of fidelity to core functional elements with adaptations to ensure compatibility with new contexts.

OBJECTIVE

This study aims to develop and apply a framework to guide tailoring and implementing CDS across diverse clinical settings.

METHODS

In preparation for a multisite trial implementing CDS for pediatric overweight or obesity in primary care, we developed the User-Centered Framework for Implementation of Technology (UFIT), a framework that integrates principles from user-centered design (UCD), human factors/ergonomics theories, and implementation science to guide both CDS adaptation and tailoring of related implementation strategies. Our transdisciplinary study team conducted semistructured interviews with pediatric primary care clinicians and a diverse group of stakeholders from 3 health systems in the northeastern, midwestern, and southeastern United States to inform and apply the framework for our formative evaluation.

RESULTS

We conducted 41 qualitative interviews with primary care clinicians (n=21) and other stakeholders (n=20). Our workflow analysis found 3 primary ways in which clinicians interact with the electronic health record during primary care well-child visits identifying opportunities for decision support. Additionally, we identified differences in practice patterns across contexts necessitating a multiprong design approach to support a variety of workflows, user needs, preferences, and implementation strategies.

CONCLUSIONS

UFIT integrates theories and guidance from UCD, human factors/ergonomics, and implementation science to promote fit with local contexts for optimal outcomes. The components of UFIT were used to guide the development of Improving Pediatric Obesity Practice Using Prompts, an integrated package comprising CDS for obesity or overweight treatment with tailored implementation strategies.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05627011; https://clinicaltrials.gov/study/NCT05627011.

Sufficiency for PSS tracking gait disorders in multiple sclerosis: A managerial perspective

This study primarily aimed to explore the capabilities of digitalisation in the healthcare context, focusing on a specific disease. In this case, the study examined the potential of remote monitoring of gait to address the sensitivity of multiple sclerosis progression to gait characteristics by adopting a non-invasive approach to remotely quantify gait disturbances in a patient's daily life. To better understand the managerial aspects associated with this approach, the researchers conducted a literature review along with a set of semi-structured interviews. The target population included MS patients as well as the key agents involved in their care: patients' family members, neurologists, MS nurses, physiotherapists, medical directors, and pharmacist. The study identifies the perceived barriers and drivers that could contribute to the successful deployment of PSS remote gait monitoring as a healthcare service: i) At mega-level governance. Implications on privacy and security data are notable barriers missing on the speech. ii) At macro level, funding is highlighted as main barrier. The cost and lack of health system subsidies may render initiatives unsustainable, as emphasised by the interviewees. iii) At meso level, useable data is recognised as a driver. The data collection process can align with diverse interests to create value and business opportunities for the ecosystem actors, enhance care, attract stakeholders, such as insurers and pharma, and form partnerships. iv) At micro-level processes, we find two potential barriers: wearable device and app usability (comfort, navigation, efficiency) and organisational/behavioural aspects (training, digital affinity, skills), which are crucial for value creation in innovation ecosystems among patients and healthcare professionals. Finally, we find an interesting gap in the literature and interviews. Stakeholders' limited awareness of technological demands, especially from information technologies, for a successful long-term service, can be consider two key barriers for PSS.

Academic Detailing as a Health Information Technology Implementation Method: Supporting the Design and Implementation of an Emergency Department-Based Clinical Decision Support Tool to Prevent Future Falls

BACKGROUND

Clinical decision support (CDS) tools that incorporate machine learning-derived content have the potential to transform clinical care by augmenting clinicians' expertise. To realize this potential, such tools must be designed to fit the dynamic work systems of the clinicians who use them. We propose the use of academic detailing-personal visits to clinicians by an expert in a specific health IT tool-as a method for both ensuring the correct understanding of that tool and its evidence base and identifying factors influencing the tool's implementation.

OBJECTIVE

This study aimed to assess academic detailing as a method for simultaneously ensuring the correct understanding of an emergency department-based CDS tool to prevent future falls and identifying factors impacting clinicians' use of the tool through an analysis of the resultant qualitative data.

METHODS

Previously, our team designed a CDS tool to identify patients aged 65 years and older who are at the highest risk of future falls and prompt an interruptive alert to clinicians, suggesting the patient be referred to a mobility and falls clinic for an evidence-based preventative intervention. We conducted 10-minute academic detailing interviews (n=16) with resident emergency medicine physicians and advanced practice providers who had encountered our CDS tool in practice. We conducted an inductive, team-based content analysis to identify factors that influenced clinicians' use of the CDS tool.

RESULTS

The following categories of factors that impacted clinicians' use of the CDS were identified: (1) aspects of the CDS tool's design (2) clinicians' understanding (or misunderstanding) of the CDS or referral process, (3) the busy nature of the emergency department environment, (4) clinicians' perceptions of the patient and their associated fall risk, and (5) the opacity of the referral process. Additionally, clinician education was done to address any misconceptions about the CDS tool or referral process, for example, demonstrating how simple it is to place a referral via the CDS and clarifying which clinic the referral goes to.

CONCLUSIONS

Our study demonstrates the use of academic detailing for supporting the implementation of health information technologies, allowing us to identify factors that impacted clinicians' use of the CDS while concurrently educating clinicians to ensure the correct understanding of the CDS tool and intervention. Thus, academic detailing can inform both real-time adjustments of a tool's implementation, for example, refinement of the language used to introduce the tool, and larger scale redesign of the CDS tool to better fit the dynamic work environment of clinicians.

Evaluating the use of clinical decision aids in an Australian emergency department: A cross-sectional survey

OBJECTIVE

To identify healthcare professionals' knowledge, self-reported use, and documentation of clinical decision aids (CDAs) in a large ED in Australia, to identify behavioural determinants influencing the use of CDAs, and healthcare professionals preferences for integrating CDAs into the electronic medical record (EMR) system.

METHODS

Healthcare professionals (doctors, nurses and physiotherapists) working in the ED at the Gold Coast Hospital, Queensland were invited to complete an online survey. Quantitative data were analysed using descriptive statistics, and where appropriate, mapped to the theoretical domains framework to identify potential barriers to the use of CDAs. Qualitative data were analysed using content analysis.

RESULTS

Seventy-four healthcare professionals (34 medical officers, 31 nurses and nine physiotherapists) completed the survey. Healthcare professionals' knowledge and self-reported use of 21 validated CDAs was low but differed considerably across CDAs. Only 4 out of 21 CDAs were reported to be used 'sometimes' or 'always' by the majority of respondents (Ottawa Ankle Rule for ankle injury, Wells' criteria for pulmonary embolism, Wells' criteria for deep vein thrombosis and PERC rule for pulmonary embolism). Most respondents wanted to increase their use of valid and reliable CDAs and supported the integration of CDAs into the EMR to facilitate their use and support documentation. Potential barriers impacting the use of CDAs represented three theoretical domains of knowledge, social/professional role and identity, and social influences.

CONCLUSIONS

CDAs are used variably by healthcare professionals and are inconsistently applied in the clinical encounter. Preferences of healthcare professionals need to be considered to allow the successful integration of CDAs into the EMR.

ROAD2H: Development and evaluation of an open-source explainable artificial intelligence approach for managing co-morbidity and clinical guidelines

Introduction

Clinical decision support (CDS) systems (CDSSs) that integrate clinical guidelines need to reflect real-world co-morbidity. In patient-specific clinical contexts, transparent recommendations that allow for contraindications and other conflicts arising from co-morbidity are a requirement. In this work, we develop and evaluate a non-proprietary, standards-based approach to the deployment of computable guidelines with explainable argumentation, integrated with a commercial electronic health record (EHR) system in Serbia, a middle-income country in West Balkans.

Methods

We used an ontological framework, the Transition-based Medical Recommendation (TMR) model, to represent, and reason about, guideline concepts, and chose the 2017 International global initiative for chronic obstructive lung disease (GOLD) guideline and a Serbian hospital as the deployment and evaluation site, respectively. To mitigate potential guideline conflicts, we used a TMR-based implementation of the Assumptions-Based Argumentation framework extended with preferences and Goals (ABA+G). Remote EHR integration of computable guidelines was via a microservice architecture based on HL7 FHIR and CDS Hooks. A prototype integration was developed to manage chronic obstructive pulmonary disease (COPD) with comorbid cardiovascular or chronic kidney diseases, and a mixed-methods evaluation was conducted with 20 simulated cases and five pulmonologists.

Results

Pulmonologists agreed 97% of the time with the GOLD-based COPD symptom severity assessment assigned to each patient by the CDSS, and 98% of the time with one of the proposed COPD care plans. Comments were favourable on the principles of explainable argumentation; inclusion of additional co-morbidities was suggested in the future along with customisation of the level of explanation with expertise.

Conclusion

An ontological model provided a flexible means of providing argumentation and explainable artificial intelligence for a long-term condition. Extension to other guidelines and multiple co-morbidities is needed to test the approach further.

Clinical Decision Support Requirements for Ventricular Tachycardia Diagnosis Within the Frameworks of Knowledge and Practice: Survey Study

BACKGROUND

Ventricular tachycardia (VT) diagnosis is challenging due to the similarity between VT and some forms of supraventricular tachycardia, complexity of clinical manifestations, heterogeneity of underlying diseases, and potential for life-threatening hemodynamic instability. Clinical decision support systems (CDSSs) have emerged as promising tools to augment the diagnostic capabilities of cardiologists. However, a requirements analysis is acknowledged to be vital for the success of a CDSS, especially for complex clinical tasks such as VT diagnosis.

OBJECTIVE

The aims of this study were to analyze the requirements for a VT diagnosis CDSS within the frameworks of knowledge and practice and to determine the clinical decision support (CDS) needs.

METHODS

Our multidisciplinary team first conducted semistructured interviews with seven cardiologists related to the clinical challenges of VT and expected decision support. A questionnaire was designed by the multidisciplinary team based on the results of interviews. The questionnaire was divided into four sections: demographic information, knowledge assessment, practice assessment, and CDS needs. The practice section consisted of two simulated cases for a total score of 10 marks. Online questionnaires were disseminated to registered cardiologists across China from December 2022 to February 2023. The scores for the practice section were summarized as continuous variables, using the mean, median, and range. The knowledge and CDS needs sections were assessed using a 4-point Likert scale without a neutral option. Kruskal-Wallis tests were performed to investigate the relationship between scores and practice years or specialty.

RESULTS

Of the 687 cardiologists who completed the questionnaire, 567 responses were eligible for further analysis. The results of the knowledge assessment showed that 383 cardiologists (68%) lacked knowledge in diagnostic evaluation. The overall average score of the practice assessment was 6.11 (SD 0.55); the etiological diagnosis section had the highest overall scores (mean 6.74, SD 1.75), whereas the diagnostic evaluation section had the lowest scores (mean 5.78, SD 1.19). A majority of cardiologists (344/567, 60.7%) reported the need for a CDSS. There was a significant difference in practice competency scores between general cardiologists and arrhythmia specialists (P=.02).

CONCLUSIONS

There was a notable deficiency in the knowledge and practice of VT among Chinese cardiologists. Specific knowledge and practice support requirements were identified, which provide a foundation for further development and optimization of a CDSS. Moreover, it is important to consider clinicians' specialization levels and years of practice for effective and personalized support.

Factors Influencing Medication Administration Outcomes Among New Graduate Nurses Using Bar Code-Assisted Medication Administration

Paramount to patient safety is the ability for nurses to make clinical decisions free from human error. Yet, the dynamic clinical environment in which nurses work is characterized by uncertainty, urgency, and high consequence, necessitating that nurses make quick and critical decisions. The aim of this study was to examine the influence of human and environmental factors on the decision to administer among new graduate nurses in response to alert generation during bar code-assisted medication administration. The design for this study was a descriptive, longitudinal, observational cohort design using EHR audit log and administrative data. The study was set at a large, urban medical center in the United States and included 132 new graduate nurses who worked on adult, inpatient units. Research variables included human and environmental factors. Data analysis included descriptive and inferential analyses. This study found that participants continued with administration of a medication in 90.75% of alert encounters. When considering the response to an alert, residency cohort, alert category, and previous exposure variables were associated with the decision to proceed with administration. It is important to continue to study factors that influence nurses' decision-making, particularly during the process of medication administration, to improve patient safety and outcomes.

Feasibility, acceptability and impact of a clinical decision support tool among primary care providers in an urban, rural and remote site in the Philippines

INTRODUCTION

Strengthening primary care helps address health inequities that continue to persist in the Philippines. The Philippine Primary Care Studies pilot-tested interventions to improve the primary care system. One intervention was the provision of a free subscription to an electronic decision support application called UpToDate (UTD) for primary care providers (PCPs), including doctors, nurses, midwives and community health workers (CHWs). This study aimed to (1) assess the feasibility of using UTD as information source for PCPs in urban, rural and remote settings, (2) determine the acceptability of UTD as an information source for PCPs and (3) examine the impact of UTD access on PCP clinical decision-making.

METHODS

Four focus group discussions (FGDs) and two key informant interviews (KII) were conducted to gather insights from 30 PCPs. Thematic analysis through coding in NVivo V.12 was done using the technology acceptance model (TAM) as a guiding framework.

RESULTS

All PCPs had positive feedback regarding UTD use because of its comprehensiveness, accessibility, mobility and general design. The participants relayed UTD's benefit for point-of-contact use, capacity-building and continuing professional development. PCPs across the three sites, including CHWs with no formal medical education, were able to provide evidence-based medical advice to patients through UTD. However, external factors in these settings impeded the full integration of UTD in the PCPs' workflow, including poor internet access, unstable sources of electricity, lack of compatible mobile devices and the need for translation to the local language.

CONCLUSION

UTD was a feasible and acceptable clinical decision support tool for the PCPs. Factors affecting the feasibility of using UTD include technological and environmental factors (ie, internet access and the lack of translation to the local language), as well as the organisational structure of the primary care facility which determines the roles of the PCPs. Despite the difference in roles and responsibilities of the PCPs, UTD positively impacted decision-making and patient education for all PCPs through its use as a point-of-contact tool and a tool for capacity-building.

Revolutionary Potential of ChatGPT in Constructing Intelligent Clinical Decision Support Systems

Recently, Chatbot Generative Pre-trained Transformer (ChatGPT) is recognized as a promising clinical decision support system (CDSS) in the medical field owing to its advanced text analysis capabilities and interactive design. However, ChatGPT primarily focuses on learning text semantics rather than learning complex data structures and conducting real-time data analysis, which typically necessitate the development of intelligent CDSS employing specialized machine learning algorithms. Although ChatGPT cannot directly execute specific algorithms, it aids in algorithm design for intelligent CDSS at the textual level. In this study, besides discussing the types of CDSS and their relationship with ChatGPT, we mainly investigate the benefits and drawbacks of employing ChatGPT as an auxiliary design tool for intelligent CDSS. Our findings indicate that by collaborating with human expertise, ChatGPT has the potential to revolutionize the development of robust and effective intelligent CDSS.

Evaluation of clinical practice guideline-derived clinical decision support systems using a novel quality model

Over the last decade, clinical practice guidelines (CPGs) have become an important asset for daily life in healthcare organizations. Efficient management and digitization of CPGs help achieve organizational objectives and improve patient care and healthcare quality by reducing variability. However, digitizing CPGs is a difficult, complex task because they are usually expressed as text, and this often leads to the development of partial software solutions. At present, different research proposals and CPG-derived CDSS (clinical decision support system) do exist for managing CPG digitalization lifecycles (from modeling to deployment and execution), but they do not all provide full lifecycle support, making it more difficult to choose solutions or proposals that fully meet the needs of a healthcare organization. This paper proposes a method based on quality models to uniformly compare and evaluate technological tools, providing a rigorous method that uses qualitative and quantitative analysis of technological aspects. In addition, this paper also presents how this method has been instantiated to evaluate and compare CPG-derived CDSS by highlighting each phase of the CPG digitization lifecycle. Finally, discussion and analysis of currently available tools are presented, identifying gaps and limitations.

Association of red blood cell distribution width with hospital admission and in-hospital mortality across all-cause adult emergency department visits

Objectives

To test the association between the initial red blood cell distribution width (RDW) value in the emergency department (ED) and hospital admission and, among those admitted, in-hospital mortality.

Materials and Methods

We perform a retrospective analysis of 210 930 adult ED visits with complete blood count results from March 2013 to February 2022. Primary outcomes were hospital admission and in-hospital mortality. Variables for each visit included demographics, comorbidities, vital signs, basic metabolic panel, complete blood count, and final diagnosis. The association of each outcome with the initial RDW value was calculated across 3 age groups (<45, 45-65, and >65) as well as across 374 diagnosis categories. Logistic regression (LR) and XGBoost models using all variables excluding final diagnoses were built to test whether RDW was a highly weighted and informative predictor for each outcome. Finally, simplified models using only age, sex, and vital signs were built to test whether RDW had additive predictive value.

Results

Compared to that of discharged visits (mean [SD]: 13.8 [2.03]), RDW was significantly elevated in visits that resulted in admission (15.1 [2.72]) and, among admissions, those resulting in intensive care unit stay (15.3 [2.88]) and/or death (16.8 [3.25]). This relationship held across age groups as well as across various diagnosis categories. An RDW >16 achieved 90% specificity for hospital admission, while an RDW >18.5 achieved 90% specificity for in-hospital mortality. LR achieved a test area under the curve (AUC) of 0.77 (95% confidence interval [CI] 0.77-0.78) for hospital admission and 0.85 (95% CI 0.81-0.88) for in-hospital mortality, while XGBoost achieved a test AUC of 0.90 (95% CI 0.89-0.90) for hospital admission and 0.96 (95% CI 0.94-0.97) for in-hospital mortality. RDW had high scaled weights and information gain for both outcomes and had additive value in simplified models predicting hospital admission.

Discussion

Elevated RDW, previously associated with mortality in myocardial infarction, pulmonary embolism, heart failure, sepsis, and COVID-19, is associated with hospital admission and in-hospital mortality across all-cause adult ED visits. Used alone, elevated RDW may be a specific, but not sensitive, test for both outcomes, with multivariate LR and XGBoost models showing significantly improved test characteristics.

Conclusions

RDW, a component of the complete blood count panel routinely ordered as the initial workup for the undifferentiated patient, may be a generalizable biomarker for acuity in the ED.

Modelling clinical narrative as computable knowledge: The NICE computable implementation guidance project

Introduction

Translating narrative clinical guidelines to computable knowledge is a long-standing challenge that has seen a diverse range of approaches. The UK National Institute for Health and Care Excellence (NICE) Content Advisory Board (CAB) aims ultimately to (1) guide clinical decision support and other software developers to increase traceability, fidelity and consistency in supporting clinical use of NICE recommendations, (2) guide local practice audit and intervention to reduce unwarranted variation, (3) provide feedback to NICE on how future recommendations should be developed.

Objectives

The first phase of work was to explore a range of technical approaches to transition NICE toward the production of natively digital content.

Methods

Following an initial 'collaborathon' in November 2022, the NICE Computable Implementation Guidance project (NCIG) was established. We held a series of workstream calls approximately fortnightly, focusing on (1) user stories and trigger events, (2) information model and definitions, (3) horizon-scanning and output format. A second collaborathon was held in March 2023 to consolidate progress across the workstreams and agree residual actions to complete.

Results

While we initially focussed on technical implementation standards, we decided that an intermediate logical model was a more achievable first step in the journey from narrative to fully computable representation. NCIG adopted the WHO Digital Adaptation Kit (DAK) as a technology-agnostic method to model user scenarios, personae, processes and workflow, core data elements and decision-support logic. Further work will address indicators, such as prescribing compliance, and implementation in document templates for primary care patient record systems.

Conclusions

The project has shown that the WHO DAK, with some modification, is a promising approach to build technology-neutral logical specifications of NICE recommendations. Implementation of concurrent computable modelling by multidisciplinary teams during guideline development poses methodological and cultural questions that are complex but tractable given suitable will and leadership.

Targeted messaging to improve the adoption of clinical decision support for prescription drug monitoring program use

Clinical decision support (CDS) can prevent medical errors and improve patient outcomes. Electronic health record (EHR)-based CDS, designed to facilitate prescription drug monitoring program (PDMP) review, has reduced inappropriate opioid prescribing. However, the pooled effectiveness of CDS has exhibited substantial heterogeneity and current literature does not adequately detail why certain CDS are more successful than others. Clinicians regularly override CDS, limiting its impact. No studies recommend how to help nonadopters recognize and recover from CDS misuse. We hypothesized that a targeted educational intervention would improve CDS adoption and effectiveness for nonadopters. Over 10 months, we identified 478 providers consistently overriding CDS (nonadopters) and sent each up to 3 educational message(s) via email or EHR-based chat. One hundred sixty-one (34%) nonadopters stopped consistently overriding CDS and started reviewing the PDMP after contact. We concluded that targeted messaging is a low-resource way to disseminate CDS education and improve CDS adoption and best practice delivery.

An Integrated Process for Co-Developing and Implementing Written and Computable Clinical Practice Guidelines

The goal of this article is to describe an integrated parallel process for the co-development of written and computable clinical practice guidelines (CPGs) to accelerate adoption and increase the impact of guideline recommendations in clinical practice. From February 2018 through December 2021, interdisciplinary work groups were formed after an initial Kaizen event and using expert consensus and available literature, produced a 12-phase integrated process (IP). The IP includes activities, resources, and iterative feedback loops for developing, implementing, disseminating, communicating, and evaluating CPGs. The IP incorporates guideline standards and informatics practices and clarifies how informaticians, implementers, health communicators, evaluators, and clinicians can help guideline developers throughout the development and implementation cycle to effectively co-develop written and computable guidelines. More efficient processes are essential to create actionable CPGs, disseminate and communicate recommendations to clinical end users, and evaluate CPG performance. Pilot testing is underway to determine how this IP expedites the implementation of CPGs into clinical practice and improves guideline uptake and health outcomes.

A lifecycle framework illustrates eight stages necessary for realizing the benefits of patient-centered clinical decision support

The design, development, implementation, use, and evaluation of high-quality, patient-centered clinical decision support (PC CDS) is necessary if we are to achieve the quintuple aim in healthcare. We developed a PC CDS lifecycle framework to promote a common understanding and language for communication among researchers, patients, clinicians, and policymakers. The framework puts the patient, and/or their caregiver at the center and illustrates how they are involved in all the following stages: Computable Clinical Knowledge, Patient-specific Inference, Information Delivery, Clinical Decision, Patient Behaviors, Health Outcomes, Aggregate Data, and patient-centered outcomes research (PCOR) Evidence. Using this idealized framework reminds key stakeholders that developing, deploying, and evaluating PC-CDS is a complex, sociotechnical challenge that requires consideration of all 8 stages. In addition, we need to ensure that patients, their caregivers, and the clinicians caring for them are explicitly involved at each stage to help us achieve the quintuple aim.

Effectiveness of an Emergency Department-Based Machine Learning Clinical Decision Support Tool to Prevent Outpatient Falls Among Older Adults: Protocol for a Quasi-Experimental Study

BACKGROUND

Emergency department (ED) providers are important collaborators in preventing falls for older adults because they are often the first health care providers to see a patient after a fall and because at-home falls are often preceded by previous ED visits. Previous work has shown that ED referrals to falls interventions can reduce the risk of an at-home fall by 38%. Screening patients at risk for a fall can be time-consuming and difficult to implement in the ED setting. Machine learning (ML) and clinical decision support (CDS) offer the potential of automating the screening process. However, it remains unclear whether automation of screening and referrals can reduce the risk of future falls among older patients.

OBJECTIVE

The goal of this paper is to describe a research protocol for evaluating the effectiveness of an automated screening and referral intervention. These findings will inform ongoing discussions about the use of ML and artificial intelligence to augment medical decision-making.

METHODS

To assess the effectiveness of our program for patients receiving the falls risk intervention, our primary analysis will be to obtain referral completion rates at 3 different EDs. We will use a quasi-experimental design known as a sharp regression discontinuity with regard to intent-to-treat, since the intervention is administered to patients whose risk score falls above a threshold. A conditional logistic regression model will be built to describe 6-month fall risk at each site as a function of the intervention, patient demographics, and risk score. The odds ratio of a return visit for a fall and the 95% CI will be estimated by comparing those identified as high risk by the ML-based CDS (ML-CDS) and those who were not but had a similar risk profile.

RESULTS

The ML-CDS tool under study has been implemented at 2 of the 3 EDs in our study. As of April 2023, a total of 1326 patient encounters have been flagged for providers, and 339 unique patients have been referred to the mobility and falls clinic. To date, 15% (45/339) of patients have scheduled an appointment with the clinic.

CONCLUSIONS

This study seeks to quantify the impact of an ML-CDS intervention on patient behavior and outcomes. Our end-to-end data set allows for a more meaningful analysis of patient outcomes than other studies focused on interim outcomes, and our multisite implementation plan will demonstrate applicability to a broad population and the possibility to adapt the intervention to other EDs and achieve similar results. Our statistical methodology, regression discontinuity design, allows for causal inference from observational data and a staggered implementation strategy allows for the identification of secular trends that could affect causal associations and allow mitigation as necessary.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05810064; https://www.clinicaltrials.gov/study/NCT05810064.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/48128.

Adaptation and Formative Evaluation of Online Decision Support to Implement Evidence-Based Strategies to Increase HPV Vaccination Rates in Pediatric Clinics

Human papilloma virus (HPV) vaccination rates remain below national goals in the United States despite the availability of evidence-based strategies to increase rates. The Adolescent Vaccination Program (AVP) is a multi-component intervention demonstrated to increase HPV vaccination rates in pediatric clinics through the implementation of six evidence-based strategies. The purpose of this study, conducted in Houston, Texas, from 2019-2021, was to adapt the AVP into an online decision support implementation tool for standalone use and to evaluate its feasibility for use in community clinics. Phase 1 (Adaptation) comprised clinic interviews (n = 23), literature review, Adolescent Vaccination Program Implementation Tool (AVP-IT) design documentation, and AVP-IT development. Phase 2 (Evaluation) comprised usability testing with healthcare providers (HCPs) (n = 5) and feasibility testing in community-based clinics (n = 2). AVP-IT decision support provides an Action Plan with tailored guidance on implementing six evidence-based strategies (immunization champions, assessment and feedback, continuing education, provider prompts, parent reminders, and parent education). HCPs rated the AVP-IT as acceptable, credible, easy, helpful, impactful, and appealing (≥80% agreement). They rated AVP-IT supported implementation as easier and more effective compared to usual practice (p ≤ 0.05). The clinic-based AVP-IT uses facilitated strategy implementation by 3-month follow-up. The AVP-IT promises accessible, utilitarian, and scalable decision support on strategies to increase HPV vaccination rates in pediatric clinic settings. Further feasibility and efficacy testing is indicated.

Where Do Real-Time Prescription Benefit Tools Fit in the Landscape of High US Prescription Medication Costs? A Narrative Review

The problem of unaffordable prescription medications in the United States is complex and can result in poor patient adherence to therapy, worse clinical outcomes, and high costs to the healthcare system. While providers are aware of the financial burden of healthcare for patients, there is a lack of actionable price transparency at the point of prescribing. Real-time prescription benefit (RTPB) tools are new electronic clinical decision support tools that retrieve patient- and medication-specific out-of-pocket cost information and display it to clinicians at the point of prescribing. The rise in US healthcare costs has been a major driver for efforts to increase medication price transparency, and mandates from the Centers for Medicare & Medicaid Services for Medicare Part D sponsors to adopt RTPB tools may spur integration of such tools into electronic health records. Although multiple factors affect the implementation of RTPB tools, there is limited evidence on outcomes. Further research will be needed to understand the impact of RTPB tools on end results such as prescribing behavior, out-of-pocket medication costs for patients, and adherence to pharmacologic treatment. We review the terminology and concepts essential in understanding the landscape of RTPB tools, implementation considerations, barriers to adoption, and directions for future research that will be important to patients, prescribers, health systems, and insurers.

Clinician interest in clinical decision support for PSA-based prostate cancer screening

OBJECTIVE

To evaluate the interest of primary care clinicians in utilizing CDS for PSA screening. Evidence suggests that electronic clinical decision support (CDS) may decrease low-value prostate-specific antigen (PSA) testing. However, physician attitudes towards CDS for PSA screening are largely unknown.

METHODS

A survey was sent to 201 primary care clinicians, including both physicians and Advanced Practice Providers (APP), within a large academic health system. Eligible clinicians cared for male patients aged 40 to 80 years and ordered ≥5 PSA tests in the past year. Respondents were stratified into 3 groups, appropriate screeners, low-value screeners, or rare-screeners, based on responses to survey questions assessing PSA screening practices. The degree of interest in electronic CDS was determined via a composite Likert score comprising relevant survey items.

RESULTS

Survey response rate was 29% (59/201) consisting of 85% MD/DO and 15% APP respondents. All clinicians surveyed were interested in CDS (P < 0.001) without significant difference between screener groups. Clinicians agreed most uniformly that CDS be evidence-based. Clinicians disagreed on whether CDS would decrease professional discretion over patient decisions.

CONCLUSIONS

Primary care clinicians are interested in CDS for PSA screening regardless of their current screening practices. Prioritizing CDS features that clinicians value, such as ensuring CDS recommendations are evidence-based, may increase the likelihood of successful implementation, whereas perceived threat to autonomy may be a hinderance to utilization.

APLUS: A Python library for usefulness simulations of machine learning models in healthcare

Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.

Management Opportunities and Challenges After Achieving Widespread Health System Digitization

The adoption of electronic health records (EHRs) and digitization of health data over the past decade is ushering in the next generation of digital health tools that leverage artificial intelligence (AI) to improve varied aspects of health system performance. The decade ahead is therefore shaping up to be one in which digital health becomes even more at the forefront of health care delivery - demanding the time, attention, and resources of health care leaders and frontline staff, and becoming inextricably linked with all dimensions of health care delivery. In this chapter, we look back and look ahead. There are substantive lessons learned from the first era of large-scale adoption of enterprise EHRs and ongoing challenges that organizations are wrestling with - particularly related to the tension between standardization and flexibility/customization of EHR systems and the processes they support. Managing this tension during efforts to implement and optimize enterprise systems is perhaps the core challenge of the past decade, and one that has impeded consistent realization of value from initial EHR investments. We describe these challenges, how they manifest, and organizational strategies to address them, with a specific focus on alignment with broader value-based care transformation. We then look ahead to the AI wave - the massive number of applications of AI to health care delivery, the expected benefits, the risks and challenges, and approaches that health systems can consider to realize the benefits while avoiding the risks.

Barriers and enablers to implementing and using clinical decision support systems for chronic diseases: a qualitative systematic review and meta-aggregation

Background

Clinical decision support (CDS) is increasingly used to facilitate chronic disease care. Despite increased availability of electronic health records and the ongoing development of new CDS technologies, uptake of CDS into routine clinical settings is inconsistent. This qualitative systematic review seeks to synthesise healthcare provider experiences of CDS—exploring the barriers and enablers to implementing, using, evaluating, and sustaining chronic disease CDS systems.

Methods

A search was conducted in Medline, CINAHL, APA PsychInfo, EconLit, and Web of Science from 2011 to 2021. Primary research studies incorporating qualitative findings were included if they targeted healthcare providers and studied a relevant chronic disease CDS intervention. Relevant CDS interventions were electronic health record-based and addressed one or more of the following chronic diseases: cardiovascular disease, diabetes, chronic kidney disease, hypertension, and hypercholesterolaemia. Qualitative findings were synthesised using a meta-aggregative approach.

Results

Thirty-three primary research articles were included in this qualitative systematic review. Meta-aggregation of qualitative data revealed 177 findings and 29 categories, which were aggregated into 8 synthesised findings. The synthesised findings related to clinical context, user, external context, and technical factors affecting CDS uptake. Key barriers to uptake included CDS systems that were simplistic, had limited clinical applicability in multimorbidity, and integrated poorly into existing workflows. Enablers to successful CDS interventions included perceived usefulness in providing relevant clinical knowledge and structured chronic disease care; user confidence gained through training and post training follow-up; external contexts comprised of strong clinical champions, allocated personnel, and technical support; and CDS technical features that are both highly functional, and attractive.

Conclusion

This systematic review explored healthcare provider experiences, focussing on barriers and enablers to CDS use for chronic diseases. The results provide an evidence-base for designing, implementing, and sustaining future CDS systems. Based on the findings from this review, we highlight actionable steps for practice and future research.

Trial registration

PROSPERO CRD42020203716

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00326-x.

Business Process Model and Notation and openEHR Task Planning for Clinical Pathway Standards in Infections: Critical Analysis

Background

Clinical pathways (CPs) are usually expressed by means of workflow formalisms, providing health care personnel with an easy-to-understand, high-level conceptual model of medical steps in specific patient conditions, thereby improving overall health care process quality in clinical practice. From a standardized perspective, the business process model and notation (BPMN), a widely spread general-purpose process formalism, has been used for conceptual modeling in clinical domains, mainly because of its easy-to-use graphical notation, facilitating the common understanding and communication of the parties involved in health care. However, BPMN is not particularly oriented toward the peculiarities of complex clinical processes such as infection diagnosis and treatment, in which time plays a critical role, which is why much of the BPMN clinical-oriented research has revolved around how to extend the standard to address these special needs. The shift from an agnostic, general-purpose BPMN notation to a natively clinical-oriented notation such as openEHR Task Planning (TP) could constitute a major step toward clinical process improvement, enhancing the representation of CPs for infection treatment and other complex scenarios.

Objective

Our work aimed to analyze the suitability of a clinical-oriented formalism (TP) to successfully represent typical process patterns in infection treatment, identifying domain-specific improvements to the standard that could help enhance its modeling capabilities, thereby promoting the widespread adoption of CPs to improve medical practice and overall health care quality.

Methods

Our methodology consisted of 4 major steps: identification of key features of infection CPs through literature review, clinical guideline analysis, and BPMN extensions; analysis of the presence of key features in TP; modeling of relevant process patterns of catheter-related bloodstream infection as a case study; and analysis and proposal of extensions in view of the results.

Results

We were able to easily represent the same logic applied in the extended BPMN-based process models in our case study using out-of-the-box standard TP primitives. However, we identified possible improvements to the current version of TP to allow for simpler conceptual models of infection CPs and possibly of other complex clinical scenarios.

Conclusions

Our study showed that the clinical-oriented TP specification is able to successfully represent the most complex catheter-related bloodstream infection process patterns depicted in our case study and identified possible extensions that can help increase its adequacy for modeling infection CPs and possibly other complex clinical conditions.

Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system

While a growing number of machine learning (ML) systems have been deployed in clinical settings with the promise of improving patient care, many have struggled to gain adoption and realize this promise. Based on a qualitative analysis of coded interviews with clinicians who use an ML-based system for sepsis, we found that, rather than viewing the system as a surrogate for their clinical judgment, clinicians perceived themselves as partnering with the technology. Our findings suggest that, even without a deep understanding of machine learning, clinicians can build trust with an ML system through experience, expert endorsement and validation, and systems designed to accommodate clinicians’ autonomy and support them across their entire workflow.

Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing

Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments at early time points, which is critical for improving sepsis outcomes. In view of the increasing use of such systems, better understanding of how they are adopted and used by healthcare providers is needed. Here, we analyzed provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System), which was deployed at five hospitals over a 2-year period. Among 9,805 retrospectively identified sepsis cases, the early detection tool achieved high sensitivity (82% of sepsis cases were identified) and a high rate of adoption: 89% of all alerts by the system were evaluated by a physician or advanced practice provider and 38% of evaluated alerts were confirmed by a provider. Adjusting for patient presentation and severity, patients with sepsis whose alert was confirmed by a provider within 3 h had a 1.85-h (95% CI 1.66-2.00) reduction in median time to first antibiotic order compared to patients with sepsis whose alert was either dismissed, confirmed more than 3 h after the alert or never addressed in the system. Finally, we found that emergency department providers and providers who had previous interactions with an alert were more likely to interact with alerts, as well as to confirm alerts on retrospectively identified patients with sepsis. Beyond efforts to improve the performance of early warning systems, efforts to improve adoption are essential to their clinical impact and should focus on understanding providers' knowledge of, experience with and attitudes toward such systems.

System for Context-Specific Visualization of Clinical Practice Guidelines (GuLiNav): Concept and Software Implementation

Background

Clinical decision support systems often adopt and operationalize existing clinical practice guidelines leading to higher guideline availability, increased guideline adherence, and data integration. Most of these systems use an internal state-based model of a clinical practice guideline to derive recommendations but do not provide the user with comprehensive insight into the model.

Objective

Here we present a novel approach based on dynamic guideline visualization that incorporates the individual patient’s current treatment context.

Methods

We derived multiple requirements to be fulfilled by such an enhanced guideline visualization. Using business process and model notation as the representation format for computer-interpretable guidelines, a combination of graph-based representation and logical inferences is adopted for guideline processing. A context-specific guideline visualization is inferred using a business rules engine.

Results

We implemented and piloted an algorithmic approach for guideline interpretation and processing. As a result of this interpretation, a context-specific guideline is derived and visualized. Our implementation can be used as a software library but also provides a representational state transfer interface. Spring, Camunda, and Drools served as the main frameworks for implementation. A formative usability evaluation of a demonstrator tool that uses the visualization yielded high acceptance among clinicians.

Conclusions

The novel guideline processing and visualization concept proved to be technically feasible. The approach addresses known problems of guideline-based clinical decision support systems. Further research is necessary to evaluate the applicability of the approach in specific medical use cases.

Evaluating a Prototype Clinical Decision Support Tool for Chronic Pain Treatment in Primary Care

OBJECTIVES

 The Chronic Pain Treatment Tracker (Tx Tracker) is a prototype decision support tool to aid primary care clinicians when caring for patients with chronic noncancer pain. This study evaluated clinicians' perceived utility of Tx Tracker in meeting information needs and identifying treatment options, and preferences for visual design.

METHODS

 We conducted 12 semi-structured interviews with primary care clinicians from four health systems in Indiana. The interviews were conducted in two waves, with prototype and interview guide revisions after the first six interviews. The interviews included exploration of Tx Tracker using a think-aloud approach and a clinical scenario. Clinicians were presented with a patient scenario and asked to use Tx Tracker to make a treatment recommendation. Last, participants answered several evaluation questions. Detailed field notes were collected, coded, and thematically analyzed by four analysts.

RESULTS

 We identified several themes: the need for clinicians to be presented with a comprehensive patient history, the usefulness of Tx Tracker in patient discussions about treatment planning, potential usefulness of Tx Tracker for patients with high uncertainty or risk, potential usefulness of Tx Tracker in aggregating scattered information, variability in expectations about workflows, skepticism about underlying electronic health record data quality, interest in using Tx Tracker to annotate or update information, interest in using Tx Tracker to translate information to clinical action, desire for interface with visual cues for risks, warnings, or treatment options, and desire for interactive functionality.

CONCLUSION

 Tools like Tx Tracker, by aggregating key information about past, current, and potential future treatments, may help clinicians collaborate with their patients in choosing the best pain treatments. Still, the use and usefulness of Tx Tracker likely relies on continued improvement of its functionality, accurate and complete underlying data, and tailored integration with varying workflows, care team roles, and user preferences.

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Electronic Health Record Risk-Stratification Tool Reduces Venous Thromboembolism Events in Surgical Patients

Background

Venous thromboembolism is a preventable cause of morbidity and mortality after surgery. To ensure that patients receive appropriate venous thromboembolism chemoprophylaxis, a nonmandatory risk-stratification tool based on patient clinical condition was implemented through the electronic health record to stratify patient risk and recommend chemoprophylaxis. We hypothesized that implementing this tool would reduce postoperative venous thromboembolism events in general surgery as well as across all surgical services.

Methods

All adult patients undergoing inpatient surgical operations (January 2012–December 2019) at a single quaternary care center and Level 1 trauma center were abstracted from institutional electronic health record database and stratified into patients admitted before and after venous thromboembolism risk-stratification tool implementation. Bivariable analyses compared venous thromboembolism chemoprophylaxis prescription and venous thromboembolism events with implementation and screening among all surgical patients as well as in general surgery patient subset.

Results

A total of 64,377 adults underwent operations: 27,819 preimplementation and 36,558 postimplementation. A significant reduction in venous thromboembolism events occurred from pre- to post-tool implementation for all cases (0.77% vs 0.47%, P < .001). General surgery patients (n = 15,723) had a significant increase in chemoprophylaxis prescription (81.9% vs 86.0%, P < .001) and a significant reduction in venous thromboembolism events (1.41% vs 0.59%, P < .001). After tool implementation, use of extended postdischarge chemoprophylaxis was greater among general surgery patient subset than the entire patient cohort (46.7% vs 29.6%, P < .001).

Conclusion

The integration of a nonmandatory electronic health record risk-stratification tool was associated with a significant reduction in venous thromboembolism events. Extended chemoprophylaxis was prescribed in nearly half of general surgery patients at very high risk for postdischarge events.

•

Implementing an electronic VTE risk-stratification tool reduced surgical VTE events.

•

Even as a nonmandatory tool, risk stratification led to overall fewer VTE events.

•

Postoperative VTE events were reduced by 39% after the tool was integrated in EHR.

•

With the tool, general surgery had 58% less VTE events and improved prophylaxis use.

Development and Effectiveness of a Clinical Decision Support System for Pressure Ulcer Prevention Care Using Machine Learning: A Quasi-experimental Study

This study was conducted to develop and evaluate the effectiveness of a clinical decision support system for pressure ulcer prevention on clinical (performance, visual discrimination ability, and decision-making ability) and cognitive (knowledge and attitude) workflow. After developing a clinical decision support system using machine learning, a quasi-experimental study was used. Data were collected between January and April 2020. Forty-nine RNs who met the inclusion criteria and worked at seven tertiary and five secondary hospitals participated. A clinical decision support system was provided to the intervention group during the same period. Differences in outcome variables between the two groups were analyzed using t tests. The level of pressure ulcer prevention nursing performance and visual differentiation ability of skin pressure and oral mucosa pressure ulcer showed significantly greater improvement in the experimental group compared with the control group, whereas clinical decision making did not differ significantly. A clinical decision support system using machine learning was partially successful in performance of skin pressure ulcer prevention, attitude, and visual differentiation ability for skin and oral mucosa pressure ulcer prevention. These findings indicated that a clinical decision support system using machine learning needs to be implemented for pressure ulcer prevention.

Using Guideline-Based Clinical Decision Support in Oncological Multidisciplinary Team Meetings: A Prospective, Multicenter Concordance Study

Background

Multidisciplinary team meetings formulate guideline-based individual treatment plans based on patient and disease characteristics and motivate reasons for deviation. Clinical decision trees could support multidisciplinary teams to adhere more accurately to guidelines. Every clinical decision tree is tailored to a specific decision moment in a care pathway and is composed of patient and disease characteristics leading to a guideline recommendation.

Objective

This study investigated (1) the concordance between multidisciplinary team and clinical decision tree recommendations and (2) the completeness of patient and disease characteristics available during multidisciplinary team meetings to apply clinical decision trees such that it results in a guideline recommendation.

Methods

This prospective, multicenter, observational concordance study evaluated 17 selected clinical decision trees, based on the prevailing Dutch guidelines for breast, colorectal and prostate cancers. In cases with sufficient data, concordance between multidisciplinary team and clinical decision tree recommendations was classified as concordant, conditional concordant (multidisciplinary team specified a prerequisite for the recommendation) and non-concordant.

Results

Fifty-nine multidisciplinary team meetings were attended in 8 different hospitals, and 355 cases were included. For 296 cases (83.4%), all patient data were available for providing an unconditional clinical decision tree recommendation. In 59 cases (16.6%), insufficient data were available resulting in provisional clinical decision tree recommendations. From the 296 successfully generated clinical decision tree recommendations, the multidisciplinary team recommendations were concordant in 249 (84.1%) cases, conditional concordant in 24 (8.1%) cases and non-concordant in 23 (7.8%) cases of which in 7 (2.4%) cases the reason for deviation from the clinical decision tree generated guideline recommendation was not motivated.

Conclusion

The observed concordance of recommendations between multidisciplinary teams and clinical decision trees and data completeness during multidisciplinary team meetings in this study indicate a potential role for implementation of clinical decision trees to support multidisciplinary team decision-making.

Making shouldr pin imple n gneral pactice: implementing an evidence-based guideline for shoulder pain, protocol for a hybrid design stepped-wedge cluster randomised study (EASIER study)

INTRODUCTION

Research suggests that current care for shoulder pain is not in line with the best available evidence. This project aims to assess the effectiveness, cost-effectiveness and the implementation of an evidence-based guideline for shoulder pain in general practice in Norway.

METHODS AND ANALYSIS

A stepped-wedge, cluster-randomised trial with a hybrid design assessing clinical effectiveness, cost-effectiveness and the effect of the implementation strategy of a guideline-based intervention in general practice. We will recruit at least 36 general practitioners (GPs) and randomise the time of cross-over from treatment as usual to the implemented intervention. The intervention includes an educational outreach visit to the GPs, a computerised decision tool for GPs and a self-management application for patients. We will measure outcomes at patient and GP levels using self-report questionnaires, focus group interviews and register based data. The primary outcome measure is the patient-reported Shoulder Pain and Disability Index measured at 12 weeks. Secondary outcomes include the EuroQol Quality of Life Measure (EQ5D-5L), direct and indirect costs, patient's global perceived effect of treatment outcome, Pain Self-Efficacy and Brief Illness Perception Questionnaire. We will evaluate the implementation process with focus on adherence to guideline treatment. We will do a cost-minimisation analysis based on direct and selected indirect costs and a cost-utility analysis based on EQ5D-5L. We will use mixed effect models to analyse primary and secondary outcomes.

ETHICS AND DISSEMINATION

Ethics approval was granted by the Regional Committee for Medical and Health Research Ethics-South East Norway (ref. no: 2019/104). Trial results will be submitted for publication in a peer-reviewed medical journal in accordance with Consolidated Standards of Reporting Trials.

TRIAL REGISTRATION NUMBER

NCT04806191.

OUP accepted manuscript

The increasing volume and richness of healthcare data collected during routine clinical practice have not yet translated into significant numbers of actionable insights that have systematically improved patient outcomes. An evidence-practice gap continues to exist in healthcare. We contest that this gap can be reduced by assessing the use of nudge theory as part of clinical decision support systems (CDSS). Deploying nudges to modify clinician behaviour and improve adherence to guideline-directed therapy represents an underused tool in bridging the evidence-practice gap. In conjunction with electronic health records (EHRs) and newer devices including artificial intelligence algorithms that are increasingly integrated within learning health systems, nudges such as CDSS alerts should be iteratively tested for all stakeholders involved in health decision-making: clinicians, researchers, and patients alike. Not only could they improve the implementation of known evidence, but the true value of nudging could lie in areas where traditional randomized controlled trials are lacking, and where clinical equipoise and variation dominate. The opportunity to test CDSS nudge alerts and their ability to standardize behaviour in the face of uncertainty may generate novel insights and improve patient outcomes in areas of clinical practice currently without a robust evidence base.

Implementing clinical decision support for reducing women Veterans' cardiovascular risk in VA: A mixed-method, longitudinal study of context, adaptation, and uptake

Evaluations of clinical decision support (CDS) implementation often struggle to measure and explain heterogeneity in uptake over time and across settings, and to account for the impact of context and adaptation on implementation success. In 2017-2020, the EMPOWER QUERI implemented a cardiovascular toolkit using a computerized template aimed at reducing women Veterans' cardiovascular risk across five Veterans Healthcare Administration (VA) sites, using an enhanced Replicating Effective Programs (REP) implementation approach. In this study, we used longitudinal joint displays of qualitative and quantitative findings to explore (1) how contextual factors emerged across sites, (2) how the template and implementation strategies were adapted in response to contextual factors, and (3) how contextual factors and adaptations coincided with template uptake across sites and over time. We identified site structure, staffing changes, relational authority of champions, and external leadership as important contextual factors. These factors gave rise to adaptations such as splitting the template into multiple parts, pairing the template with a computerized reminder, conducting academic detailing, creating cheat sheets, and using small-scale pilot testing. All five sites exhibited variability in utilization over the months of implementation, though later sites exhibited higher template utilization immediately post-launch, possibly reflecting a "preloading" of adaptations from previous sites. These findings underscore the importance of adaptive approaches to implementation, with intentional shifts in intervention and strategy to meet the needs of individual sites, as well as the value of integrating mixed-method data sources in conducting longitudinal evaluation of implementation efforts.

Argumentation schemes for clinical decision support

This paper demonstrates how argumentation schemes can be used in decision support systems that help clinicians in making treatment decisions. The work builds on the use of computational argumentation, a rigorous approach to reasoning with complex data that places strong emphasis on being able to justify and explain the decisions that are recommended. The main contribution of the paper is to present a novel set of specialised argumentation schemes that can be used in the context of a clinical decision support system to assist in reasoning about what treatments to offer. These schemes provide a mechanism for capturing clinical reasoning in such a way that it can be handled by the formal reasoning mechanisms of formal argumentation. The paper describes how the integration between argumentation schemes and formal argumentation may be carried out, sketches how this is achieved by an implementation that we have created and illustrates the overall process on a small set of case studies.

Workflow integration analysis of a human factors-based clinical decision support in the emergency department

Numerous challenges with the implementation, acceptance, and use of health IT are related to poor usability and a lack of integration of the technologies into clinical workflow, and have, therefore, limited the potential of these technologies to improve patient safety. We propose a definition and conceptual model of health IT workflow integration. Using interviews of 12 emergency department (ED) physicians, we identify 134 excerpts of barriers and facilitators to workflow integration of a human factors (HF)-based clinical decision support (CDS) implemented in the ED. Using data on these 134 barriers and facilitators, we distinguish 25 components of workflow integration of the CDS, which are described according to four dimensions of workflow integration: time, flow, scope of patient journey, and level. The proposed definition and conceptual model of workflow integration can be used to inform health IT design; this is the purpose of the proposed checklist that can help to ensure consideration of workflow integration during the development of health IT.

Success Factors of Artificial Intelligence Implementation in Healthcare

Background: Artificial Intelligence (AI) in healthcare has demonstrated high efficiency in academic research, while only few, and predominantly small, real-world AI applications exist in the preventive, diagnostic and therapeutic contexts. Our identification and analysis of success factors for the implementation of AI aims to close the gap between recent years' significant academic AI advancements and the comparably low level of practical application in healthcare. Methods: A literature and real life cases analysis was conducted in Scopus and OpacPlus as well as the Google advanced search database. The according search queries have been defined based on success factor categories for AI implementation derived from a prior World Health Organization survey about barriers of adoption of Big Data within 125 countries. The eligible publications and real life cases were identified through a catalog of in- and exclusion criteria focused on concrete AI application cases. These were then analyzed to deduct and discuss success factors that facilitate or inhibit a broad-scale implementation of AI in healthcare. Results: The analysis revealed three categories of success factors, namely (1) policy setting, (2) technological implementation, and (3) medical and economic impact measurement. For each of them a set of recommendations has been deducted: First, a risk adjusted policy frame is required that distinguishes between precautionary and permissionless principles, and differentiates among accountability, liability, and culpability. Second, a "privacy by design" centered technology infrastructure shall be applied that enables practical and legally compliant data access. Third, the medical and economic impact need to be quantified, e.g., through the measurement of quality-adjusted life years while applying the CHEERS and PRISMA reporting criteria. Conclusions: Private and public institutions can already today leverage AI implementation based on the identified results and thus drive the translation from scientific development to real world application. Additional success factors could include trust-building measures, data categorization guidelines, and risk level assessments and as the success factors are interlinked, future research should elaborate on their optimal interaction to utilize the full potential of AI in real world application.

Pragmatic clinic randomized trial to improve chronic kidney disease care: Design and adaptation due to COVID disruptions

BACKGROUND

We describe a clinic-randomized trial to improve chronic kidney disease (CKD) care through a CKD-clinical decision support (CKD-CDS) intervention in primary care clinics and the challenges we encountered due to COVID-19 care disruption.

METHODS/DESIGN

Primary care clinics (N = 32) were randomized to usual care (UC) or to CKD-CDS. Between April 17, 2019 and March 14, 2020, more than 7000 patients had accrued for analysis by meeting study-eligibility criteria at an index office visit: age 18-75, laboratory criteria for stage 3 or 4 CKD (eGFR 15-59 mL/min/1.73 m2), and one or more opportunities algorithmically identified to improve CKD care such as blood pressure (BP) or glucose control, angiotensin converting enzyme inhibitor (ACEI) or angiotensin receptor blocker (ARB) use, discontinuance of a nonsteroidal anti-inflammatory drug (NSAID), or nephrology referral. At CKD-CDS clinics, CDS provided individualized treatment suggestions that were printed for patients and clinicians at the start of office encounters and were viewable within the electronic health record. By initial design, the impact of the CKD-CDS intervention on care gaps was to be assessed 12 months after the index date, but COVID-19 caused major disruptions to care delivery during the intervention period. In response to disruptions, the intervention was temporarily suspended while we expanded CDS use for telehealth encounters and programmed new criteria for displaying the CKD-CDS to intervention patients due to clinic closures and scheduling changes.

DISCUSSION

We describe a NIH-funded pragmatic trial of web-based EHR-integrated CKD-CDS and modifications necessary mid-study to complete the study as intended in the face of COVID-19 pandemic challenges.

New Standards for Clinical Decision Support: A Survey of The State of Implementation

Objectives: To review the current state of research on designing and implementing clinical decision support (CDS) using four current interoperability standards: Fast Healthcare Interoperability Resources (FHIR); Substitutable Medical Applications and Reusable Technologies (SMART); Clinical Quality Language (CQL); and CDS Hooks.

Methods: We conducted a review of original studies describing development of specific CDS tools or infrastructures using one of the four targeted standards, regardless of implementation stage. Citations published any time before the literature search was executed on October 21, 2020 were retrieved from PubMed. Two reviewers independently screened articles and abstracted data according to a protocol designed by team consensus.

Results: Of 290 articles identified via PubMed search, 44 were included in this study. More than three quarters were published since 2018. Forty-three (98%) used FHIR; 22 (50%) used SMART; two (5%) used CQL; and eight (18%) used CDS Hooks. Twenty-four (55%) were in the design stage, 15 (34%) in the piloting stage, and five (11%) were deployed in a real-world setting. Only 12 (27%) of the articles reported an evaluation of the technology under development. Three of the four articles describing a deployed technology reported an evaluation. Only two evaluations with randomized study components were identified.

Conclusion: The diversity of topics and approaches identified in the literature highlights the utility of these standards. The infrequency of reported evaluations, as well as the high number of studies in the design or piloting stage, indicate that these technologies are still early in their life cycles. Informaticists will require a stronger evidence base to understand the implications of using these standards in CDS design and implementation.

Social dynamics of a population-level dashboard for antimicrobial stewardship: A qualitative analysis

OBJECTIVE

To evaluate antimicrobial stewards' experiences of using a dashboard display integrating local and national antibiotic use data implemented in the U.S. Department of Veterans Affairs (VA). This paper reports early formative evaluation.

DESIGN

Qualitative interviewing.

SETTING

Eight VA hospitals participated with established antimicrobial stewardship (AS) programs participated in the pilot.

PARTICIPANTS

Six infectious disease physicians and eight clinical pharmacists agreed to be interviewed (n = 14).

METHODS

A 3-part qualitative interview script was used involving a description of local stewardship activities, a Critical Incident description of dashboard use, and general questions regarding attitudes towards the tool. An inductive open coding approach was used for analysis.

RESULTS

We found 4 themes showing the complexities of using stewardship tools: (1) Data validity is socially negotiated; (2) Performance feedback motivates and persuades social goals when situated in an empirical distribution; (3) Shared problem awareness is aided by authoritative data; and (4) The AS dashboard encourages connections with local quality improvement culture.

CONCLUSIONS

Social dimensions of AS tool use emerged as distinct from, and equally important as decision support provided by the dashboard. Successful stewardship tools should be designed to support both the social and cognitive needs of users.

Barriers and facilitators to the adoption of electronic clinical decision support systems: a qualitative interview study with UK general practitioners

Background

Well-established electronic data capture in UK general practice means that algorithms, developed on patient data, can be used for automated clinical decision support systems (CDSSs). These can predict patient risk, help with prescribing safety, improve diagnosis and prompt clinicians to record extra data. However, there is persistent evidence of low uptake of CDSSs in the clinic. We interviewed UK General Practitioners (GPs) to understand what features of CDSSs, and the contexts of their use, facilitate or present barriers to their use.

Methods

We interviewed 11 practicing GPs in London and South England using a semi-structured interview schedule and discussed a hypothetical CDSS that could detect early signs of dementia. We applied thematic analysis to the anonymised interview transcripts.

Results

We identified three overarching themes: trust in individual CDSSs; usability of individual CDSSs; and usability of CDSSs in the broader practice context, to which nine subthemes contributed. Trust was affected by CDSS provenance, perceived threat to autonomy and clear management guidance. Usability was influenced by sensitivity to the patient context, CDSS flexibility, ease of control, and non-intrusiveness. CDSSs were more likely to be used by GPs if they did not contribute to alert proliferation and subsequent fatigue, or if GPs were provided with training in their use.

Conclusions

Building on these findings we make a number of recommendations for CDSS developers to consider when bringing a new CDSS into GP patient records systems. These include co-producing CDSS with GPs to improve fit within clinic workflow and wider practice systems, ensuring a high level of accuracy and a clear clinical pathway, and providing CDSS training for practice staff. These recommendations may reduce the proliferation of unhelpful alerts that can result in important decision-support being ignored.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01557-z.

From clinical decision support to clinical reasoning support systems

Despite the great promises that artificial intelligence (AI) holds for health care, the uptake of such technologies into medical practice is slow. In this paper, we focus on the epistemological issues arising from the development and implementation of a class of AI for clinical practice, namely clinical decision support systems (CDSS). We will first provide an overview of the epistemic tasks of medical professionals, and then analyse which of these tasks can be supported by CDSS, while also explaining why some of them should remain the territory of human experts. Clinical decision making involves a reasoning process in which clinicians combine different types of information into a coherent and adequate 'picture of the patient' that enables them to draw explainable and justifiable conclusions for which they bear epistemological responsibility. Therefore, we suggest that it is more appropriate to think of a CDSS as clinical reasoning support systems (CRSS). Developing CRSS that support clinicians' reasoning process therefore requires that: (a) CRSSs are developed on the basis of relevant and well-processed data; and (b) the system facilitates an interaction with the clinician. Therefore, medical experts must collaborate closely with AI experts developing the CRSS. In addition, responsible use of an CRSS requires that the data generated by the CRSS is empirically justified through an empirical link with the individual patient. In practice, this means that the system indicates what factors contributed to arriving at an advice, allowing the user (clinician) to evaluate whether these factors are medically plausible and applicable to the patient. Finally, we defend that proper implementation of CRSS allows combining human and artificial intelligence into hybrid intelligence, were both perform clearly delineated and complementary empirical tasks. Whereas CRSSs can assist with statistical reasoning and finding patterns in complex data, it is the clinicians' task to interpret, integrate and contextualize.

Nurse Cognition, Decision Support, and Barcode Medication Administration: A Conceptual Framework for Research, Practice, and Education

This article synthesizes theoretical perspectives related to nurse cognition. We present a conceptual model that can be used by multiple stakeholders to study and contemplate how nurses use clinical decision support systems, and specifically, Barcode-Assisted Medication Administration, to make decisions during the delivery of care. Theoretical perspectives integrated into the model include dual process theory, the Cognitive Continuum Theory, human factors engineering, and the Recognition-Primed Decision model. The resulting framework illustrates the process of nurse cognition during Barcode-Assisted Medication Administration. Additionally, the model includes individual or human and environmental factors that may influence nurse cognition and decision making. It is important to consider the influence of individual, human, and environmental factors on the process of nurse cognition and decision making. Specifically, it is necessary to explore the impact of heuristics and biases on clinician decision making, particularly related to the development of alarm and alert fatigue. Aided by the proposed framework, stakeholders may begin to identify heuristics and cognitive biases that influence the decision of clinicians to accept or override a clinical decision support system alert and whether heuristics and biases are associated with inappropriate alert override.