The impact of clinical decision support systems on provider behavior in the inpatient setting: A systematic review and meta-analysis

Usability Testing of Situation Awareness Clinical Decision Support in the Intensive Care Unit

OBJECTIVE

 Our objective was to evaluate the usability of an automated clinical decision support (CDS) tool previously implemented in the pediatric intensive care unit (PICU) to promote shared situation awareness among the medical team to prevent serious safety events within children's hospitals.

METHODS

 We conducted a mixed-methods usability evaluation of a CDS tool in a PICU at a large, urban, quaternary, free-standing children's hospital in the Midwest. Quantitative assessment was done using the system usability scale (SUS), while qualitative assessment involved think-aloud usability testing. The SUS was scored according to survey guidelines. For think-aloud testing, task times were calculated, and means and standard deviations were determined, stratified by role. Qualitative feedback from participants and moderator observations were summarized.

RESULTS

 Fifty-one PICU staff members, including physicians, advanced practice providers, nurses, and respiratory therapists, completed the SUS, while ten participants underwent think-aloud usability testing. The overall median usability score was 87.5 (interquartile range: 80-95), with over 96% rating the tool's usability as "good" or "excellent." Task completion times ranged from 2 to 92 seconds, with the quickest completion for reviewing high-risk criteria and the slowest for adding to high-risk criteria. Observations and participant responses from think-aloud testing highlighted positive aspects of learnability and clear display of complex information that is easily accessed, as well as opportunities for improvement in tool integration into clinical workflows.

CONCLUSION

 The PICU Warning Tool demonstrates good usability in the critical care setting. This study demonstrates the value of postimplementation usability testing in identifying opportunities for continued improvement of CDS tools.

Using Theories, Models, and Frameworks to Inform Implementation Cycles of Computerized Clinical Decision Support Systems in Tertiary Health Care Settings: Scoping Review

BACKGROUND

Computerized clinical decision support systems (CDSSs) are essential components of modern health system service delivery, particularly within acute care settings such as hospitals. Theories, models, and frameworks may assist in facilitating the implementation processes associated with CDSS innovation and its use within these care settings. These processes include context assessments to identify key determinants, implementation plans for adoption, promoting ongoing uptake, adherence, and long-term evaluation. However, there has been no prior review synthesizing the literature regarding the theories, models, and frameworks that have informed the implementation and adoption of CDSSs within hospitals.

OBJECTIVE

This scoping review aims to identify the theory, model, and framework approaches that have been used to facilitate the implementation and adoption of CDSSs in tertiary health care settings, including hospitals. The rationales reported for selecting these approaches, including the limitations and strengths, are described.

METHODS

A total of 5 electronic databases were searched (CINAHL via EBSCOhost, PubMed, Scopus, PsycINFO, and Embase) to identify studies that implemented or adopted a CDSS in a tertiary health care setting using an implementation theory, model, or framework. No date or language limits were applied. A narrative synthesis was conducted using full-text publications and abstracts. Implementation phases were classified according to the "Active Implementation Framework stages": exploration (feasibility and organizational readiness), installation (organizational preparation), initial implementation (initiating implementation, ie, training), full implementation (sustainment), and nontranslational effectiveness studies.

RESULTS

A total of 81 records (42 full text and 39 abstracts) were included. Full-text studies and abstracts are reported separately. For full-text studies, models (18/42, 43%), followed by determinants frameworks (14/42,33%), were most frequently used to guide adoption and evaluation strategies. Most studies (36/42, 86%) did not list the limitations associated with applying a specific theory, model, or framework.

CONCLUSIONS

Models and related quality improvement methods were most frequently used to inform CDSS adoption. Models were not typically combined with each other or with theory to inform full-cycle implementation strategies. The findings highlight a gap in the application of implementation methods including theories, models, and frameworks to facilitate full-cycle implementation strategies for hospital CDSSs.

Development, Design and Utilization of a CDSS for Refeeding Syndrome in Real Life Inpatient Care-A Feasibility Study

BACKGROUND

The refeeding syndrome (RFS) is an oftentimes-unrecognized complication of reintroducing nutrition in malnourished patients that can lead to fatal cardiovascular failure. We hypothesized that a clinical decision support system (CDSS) can improve RFS recognition and management.

METHODS

We developed an algorithm from current diagnostic criteria for RFS detection, tested the algorithm on a retrospective dataset and combined the final algorithm with therapy and referral recommendations in a knowledge-based CDSS. The CDSS integration into clinical practice was prospectively investigated for six months.

RESULTS

The utilization of the RFS-CDSS lead to RFS diagnosis in 13 out of 21 detected cases (62%). It improved patient-related care and documentation, e.g., RFS-specific coding (E87.7), increased from once coded in 30 month in the retrospective cohort to four times in six months in the prospective cohort and doubled the rate of nutrition referrals in true positive patients (retrospective referrals in true positive patients 33% vs. prospective referrals in true positive patients 71%).

CONCLUSION

CDSS-facilitated RFS diagnosis is possible and improves RFS recognition. This effect and its impact on patient-related outcomes needs to be further investigated in a large randomized-controlled trial.

Construction and effectiveness evaluation of a knowledge-based infectious disease monitoring and decision support system

To improve the hospital's ability to proactively detect infectious diseases, a knowledge-based infectious disease monitoring and decision support system was established based on real medical records and knowledge rules. The effectiveness of the system was evaluated using interrupted time series analysis. In the system, a monitoring and alert rule library for infectious diseases was generated by combining infectious disease diagnosis guidelines with literature and a real medical record knowledge map. The system was integrated with the electronic medical record system, and doctors were provided with various types of real-time warning prompts when writing medical records. The effectiveness of the system's alerts was analyzed from the perspectives of false positive rates, rule accuracy, alert effectiveness, and missed case rates using interrupted time series analysis. Over a period of 12 months, the system analyzed 4,497,091 medical records, triggering a total of 12,027 monitoring alerts. Of these, 98.43% were clinically effective, while 1.56% were invalid alerts, mainly owing to the relatively rough rules generated by the guidelines leading to several false alarms. In addition, the effectiveness of the system's alerts, distribution of diagnosis times, and reporting efficiency of doctors were analyzed. 89.26% of infectious disease cases could be confirmed and reported by doctors within 5 min of receiving the alert, and 77.6% of doctors could complete the filling of 33 items of information within 2 min, which is a reduction in time compared to the past. The timely reminders from the system reduced the rate of missed cases by doctors; the analysis using interrupted time series method showed an average reduction of 4.4037% in the missed-case rate. This study proposed a knowledge-based infectious disease decision support system based on real medical records and knowledge rules, and its effectiveness was verified. The system improved the management of infectious diseases, increased the reliability of decision-making, and reduced the rate of underreporting.

Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review

BACKGROUND

Successful implementation and utilization of Computerized Clinical Decision Support Systems (CDSS) in hospitals is complex and challenging. Implementation science, and in particular the Nonadoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework, may offer a systematic approach for identifying and addressing these challenges. This review aimed to identify, categorize, and describe barriers and facilitators to CDSS implementation in hospital settings and map them to the NASSS framework. Exploring the applicability of the NASSS framework to CDSS implementation was a secondary aim.

METHODS

Electronic database searches were conducted (21 July 2020; updated 5 April 2022) in Ovid MEDLINE, Embase, Scopus, PyscInfo, and CINAHL. Original research studies reporting on measured or perceived barriers and/or facilitators to implementation and adoption of CDSS in hospital settings, or attitudes of healthcare professionals towards CDSS were included. Articles with a primary focus on CDSS development were excluded. No language or date restrictions were applied. We used qualitative content analysis to identify determinants and organize them into higher-order themes, which were then reflexively mapped to the NASSS framework.

RESULTS

Forty-four publications were included. These comprised a range of study designs, geographic locations, participants, technology types, CDSS functions, and clinical contexts of implementation. A total of 227 individual barriers and 130 individual facilitators were identified across the included studies. The most commonly reported influences on implementation were fit of CDSS with workflows (19 studies), the usefulness of the CDSS output in practice (17 studies), CDSS technical dependencies and design (16 studies), trust of users in the CDSS input data and evidence base (15 studies), and the contextual fit of the CDSS with the user's role or clinical setting (14 studies). Most determinants could be appropriately categorized into domains of the NASSS framework with barriers and facilitators in the "Technology," "Organization," and "Adopters" domains most frequently reported. No determinants were assigned to the "Embedding and Adaptation Over Time" domain.

CONCLUSIONS

This review identified the most common determinants which could be targeted for modification to either remove barriers or facilitate the adoption and use of CDSS within hospitals. Greater adoption of implementation theory should be encouraged to support CDSS implementation.

Clinician Perspectives on Barriers and Enablers to Implementing an Inpatient Oncology Early Warning System: A Mixed-Methods Study

PURPOSE

To elicit end-user and stakeholder perceptions regarding design and implementation of an inpatient clinical deterioration early warning system (EWS) for oncology patients to better fit routine clinical practices and enhance clinical impact.

METHODS

In an explanatory-sequential mixed-methods study, we evaluated a stakeholder-informed oncology early warning system (OncEWS) using surveys and semistructured interviews. Stakeholders were physicians, advanced practice providers (APPs), and nurses. For qualitative data, we used grounded theory and thematic content analysis via the constant comparative method to identify determinants of OncEWS implementation.

RESULTS

Survey respondents generally agreed that an oncology-focused EWS could add value beyond clinical judgment, with nurses endorsing this notion significantly more strongly than other clinicians (nurse: median 5 on a 6-point scale [6 = strongly agree], interquartile range 4-5; doctors/advanced practice providers: 4 [4-5]; P = .005). However, some respondents would not trust an EWS to identify risk accurately (n = 36 [42%] somewhat or very concerned), while others were concerned that institutional culture would not embrace such an EWS (n = 17 [28%]).Interviews highlighted important aspects of the EWS and the local context that might facilitate implementation, including (1) a model tailored to the subtleties of oncology patients, (2) transparent model information, and (3) nursing-centric workflows. Interviewees raised the importance of sepsis as a common and high-risk deterioration syndrome.

CONCLUSION

Stakeholders prioritized maximizing the degree to which the OncEWS is understandable, informative, actionable, and workflow-complementary, and perceived these factors to be key for translation into clinical benefit.

Development of an Interoperable and Easily Transferable Clinical Decision Support System Deployment Platform: System Design and Development Study

BACKGROUND

A clinical decision support system (CDSS) is recognized as a technology that enhances clinical efficacy and safety. However, its full potential has not been realized, mainly due to clinical data standards and noninteroperable platforms.

OBJECTIVE

In this paper, we introduce the common data model-based intelligent algorithm network environment (CANE) platform that supports the implementation and deployment of a CDSS.

METHODS

CDSS reasoning engines, usually represented as R or Python objects, are deployed into the CANE platform and converted into C# objects. When a clinician requests CANE-based decision support in the electronic health record (EHR) system, patients' information is transformed into Health Level 7 Fast Healthcare Interoperability Resources (FHIR) format and transmitted to the CANE server inside the hospital firewall. Upon receiving the necessary data, the CANE system's modules perform the following tasks: (1) the preprocessing module converts the FHIRs into the input data required by the specific reasoning engine, (2) the reasoning engine module operates the target algorithms, (3) the integration module communicates with the other institutions' CANE systems to request and transmit a summary report to aid in decision support, and (4) creates a user interface by integrating the summary report and the results calculated by the reasoning engine.

RESULTS

We developed a CANE system such that any algorithm implemented in the system can be directly called through the RESTful application programming interface when it is integrated with an EHR system. Eight algorithms were developed and deployed in the CANE system. Using a knowledge-based algorithm, physicians can screen patients who are prone to sepsis and obtain treatment guides for patients with sepsis with the CANE system. Further, using a nonknowledge-based algorithm, the CANE system supports emergency physicians' clinical decisions about optimum resource allocation by predicting a patient's acuity and prognosis during triage.

CONCLUSIONS

We successfully developed a common data model-based platform that adheres to medical informatics standards and could aid artificial intelligence model deployment using R or Python.