Workflow integration analysis of a human factors-based clinical decision support in the emergency department

Patient-centered clinical decision support challenges and opportunities identified from workflow execution models

OBJECTIVE

To use workflow execution models to highlight new considerations for patient-centered clinical decision support policies (PC CDS), processes, procedures, technology, and expertise required to support new workflows.

METHODS

To generate and refine models, we used (1) targeted literature reviews; (2) key informant interviews with 6 external PC CDS experts; (3) model refinement based on authors' experience; and (4) validation of the models by a 26-member steering committee.

RESULTS AND DISCUSSION

We identified 7 major issues that provide significant challenges and opportunities for healthcare systems, researchers, administrators, and health IT and app developers. Overcoming these challenges presents opportunities for new or modified policies, processes, procedures, technology, and expertise to: (1) Ensure patient-generated health data (PGHD), including patient-reported outcomes (PROs), are documented, reviewed, and managed by appropriately trained clinicians, between visits and after regular working hours. (2) Educate patients to use connected medical devices and handle technical issues. (3) Facilitate collection and incorporation of PGHD, PROs, patient preferences, and social determinants of health into existing electronic health records. (4) Troubleshoot erroneous data received from devices. (5) Develop dashboards to display longitudinal patient-reported data. (6) Provide reimbursement to support new models of care. (7) Support patient engagement with remote devices.

CONCLUSION

Several new policies, processes, technologies, and expertise are required to ensure safe and effective implementation and use of PC CDS. As we gain more experience implementing and working with PC CDS, we should be able to begin realizing the long-term positive impact on patient health that the patient-centered movement in healthcare promises.

Radiologists' perspectives on the workflow integration of an artificial intelligence-based computer-aided detection system: A qualitative study

In healthcare, artificial intelligence (AI) is expected to improve work processes, yet most research focuses on the technical features of AI rather than its real-world clinical implementation. To evaluate the implementation process of an AI-based computer-aided detection system (AI-CAD) for prostate MRI readings, we interviewed German radiologists in a pre-post design. We embedded our findings in the Model of Workflow Integration and the Technology Acceptance Model to analyze workflow effects, facilitators, and barriers. The most prominent barriers were: (i) a time delay in the work process, (ii) additional work steps to be taken, and (iii) an unstable performance of the AI-CAD. Most frequently named facilitators were (i) good self-organization, and (ii) good usability of the software. Our results underline the importance of a holistic approach to AI implementation considering the sociotechnical work system and provide valuable insights into key factors of the successful adoption of AI technologies in work systems.

Academic Detailing as a Health Information Technology Implementation Method: Supporting the Design and Implementation of an Emergency Department-Based Clinical Decision Support Tool to Prevent Future Falls

BACKGROUND

Clinical decision support (CDS) tools that incorporate machine learning-derived content have the potential to transform clinical care by augmenting clinicians' expertise. To realize this potential, such tools must be designed to fit the dynamic work systems of the clinicians who use them. We propose the use of academic detailing-personal visits to clinicians by an expert in a specific health IT tool-as a method for both ensuring the correct understanding of that tool and its evidence base and identifying factors influencing the tool's implementation.

OBJECTIVE

This study aimed to assess academic detailing as a method for simultaneously ensuring the correct understanding of an emergency department-based CDS tool to prevent future falls and identifying factors impacting clinicians' use of the tool through an analysis of the resultant qualitative data.

METHODS

Previously, our team designed a CDS tool to identify patients aged 65 years and older who are at the highest risk of future falls and prompt an interruptive alert to clinicians, suggesting the patient be referred to a mobility and falls clinic for an evidence-based preventative intervention. We conducted 10-minute academic detailing interviews (n=16) with resident emergency medicine physicians and advanced practice providers who had encountered our CDS tool in practice. We conducted an inductive, team-based content analysis to identify factors that influenced clinicians' use of the CDS tool.

RESULTS

The following categories of factors that impacted clinicians' use of the CDS were identified: (1) aspects of the CDS tool's design (2) clinicians' understanding (or misunderstanding) of the CDS or referral process, (3) the busy nature of the emergency department environment, (4) clinicians' perceptions of the patient and their associated fall risk, and (5) the opacity of the referral process. Additionally, clinician education was done to address any misconceptions about the CDS tool or referral process, for example, demonstrating how simple it is to place a referral via the CDS and clarifying which clinic the referral goes to.

CONCLUSIONS

Our study demonstrates the use of academic detailing for supporting the implementation of health information technologies, allowing us to identify factors that impacted clinicians' use of the CDS while concurrently educating clinicians to ensure the correct understanding of the CDS tool and intervention. Thus, academic detailing can inform both real-time adjustments of a tool's implementation, for example, refinement of the language used to introduce the tool, and larger scale redesign of the CDS tool to better fit the dynamic work environment of clinicians.

Usability of a Human Factors-based Clinical Decision Support in the Emergency Department: Lessons Learned for Design and Implementation

OBJECTIVE

To evaluate the usability and use of human factors (HF)-based clinical decision support (CDS) implemented in the emergency department (ED).

BACKGROUND

Clinical decision support can improve patient safety; however, the acceptance and use of CDS has faced challenges. Following a human-centered design process, we designed a CDS to support pulmonary embolism (PE) diagnosis in the ED. We demonstrated high usability of the CDS during scenario-based usability testing. We implemented the HF-based CDS in one ED in December 2018.

METHOD

We conducted a survey of ED physicians to evaluate the usability and use of the HF-based CDS. We distributed the survey via Qualtrics, a web-based survey platform. We compared the computer system usability questionnaire scores of the CDS between those collected in the usability testing to use of the CDS in the real environment. We asked physicians about their acceptance and use of the CDS, barriers to using the CDS, and areas for improvement.

RESULTS

Forty-seven physicians (56%) completed the survey. Physicians agreed that diagnosing PE is a major problem and risk scores can support the PE diagnostic process. Usability of the CDS was reported as high, both in the experimental setting and the real clinical setting. However, use of the CDS was low. We identified several barriers to the CDS use in the clinical environment, in particular a lack of workflow integration.

CONCLUSION

Design of CDS should be a continuous process and focus on the technology's usability in the context of the broad work system and clinician workflow.

Exploring the role of professional identity in the implementation of clinical decision support systems-a narrative review

BACKGROUND

Clinical decision support systems (CDSSs) have the potential to improve quality of care, patient safety, and efficiency because of their ability to perform medical tasks in a more data-driven, evidence-based, and semi-autonomous way. However, CDSSs may also affect the professional identity of health professionals. Some professionals might experience these systems as a threat to their professional identity, as CDSSs could partially substitute clinical competencies, autonomy, or control over the care process. Other professionals may experience an empowerment of the role in the medical system. The purpose of this study is to uncover the role of professional identity in CDSS implementation and to identify core human, technological, and organizational factors that may determine the effect of CDSSs on professional identity.

METHODS

We conducted a systematic literature review and included peer-reviewed empirical studies from two electronic databases (PubMed, Web of Science) that reported on key factors to CDSS implementation and were published between 2010 and 2023. Our explorative, inductive thematic analysis assessed the antecedents of professional identity-related mechanisms from the perspective of different health care professionals (i.e., physicians, residents, nurse practitioners, pharmacists).

RESULTS

One hundred thirty-one qualitative, quantitative, or mixed-method studies from over 60 journals were included in this review. The thematic analysis found three dimensions of professional identity-related mechanisms that influence CDSS implementation success: perceived threat or enhancement of professional control and autonomy, perceived threat or enhancement of professional skills and expertise, and perceived loss or gain of control over patient relationships. At the technological level, the most common issues were the system's ability to fit into existing clinical workflows and organizational structures, and its ability to meet user needs. At the organizational level, time pressure and tension, as well as internal communication and involvement of end users were most frequently reported. At the human level, individual attitudes and emotional responses, as well as familiarity with the system, most often influenced the CDSS implementation. Our results show that professional identity-related mechanisms are driven by these factors and influence CDSS implementation success. The perception of the change of professional identity is influenced by the user's professional status and expertise and is improved over the course of implementation.

CONCLUSION

This review highlights the need for health care managers to evaluate perceived professional identity threats to health care professionals across all implementation phases when introducing a CDSS and to consider their varying manifestations among different health care professionals. Moreover, it highlights the importance of innovation and change management approaches, such as involving health professionals in the design and implementation process to mitigate threat perceptions. We provide future areas of research for the evaluation of the professional identity construct within health care.

Prospective effects of an artificial intelligence-based computer-aided detection system for prostate imaging on routine workflow and radiologists' outcomes

OBJECTIVES

Artificial intelligence (AI) is expected to alleviate the negative consequences of rising case numbers for radiologists. Currently, systematic evaluations of the impact of AI solutions in real-world radiological practice are missing. Our study addresses this gap by investigating the impact of the clinical implementation of an AI-based computer-aided detection system (CAD) for prostate MRI reading on clinicians' workflow, workflow throughput times, workload, and stress.

MATERIALS AND METHODS

CAD was newly implemented into radiology workflow and accompanied by a prospective pre-post study design. We assessed prostate MRI case readings using standardized work observations and questionnaires. The observation period was three months each in a single department. Workflow throughput times, PI-RADS score, CAD usage and radiologists' self-reported workload and stress were recorded. Linear mixed models were employed for effect identification.

RESULTS

In data analyses, 91 observed case readings (pre: 50, post: 41) were included. Variation of routine workflow was observed following CAD implementation. A non-significant increase in overall workflow throughput time was associated with CAD implementation (mean 16.99 ± 6.21 vs 18.77 ± 9.69 min, p = .51), along with an increase in diagnostic reading time for high suspicion cases (mean 15.73 ± 4.99 vs 23.07 ± 8.75 min, p = .02). Changes in radiologists' self-reported workload or stress were not found.

CONCLUSION

Implementation of an AI-based detection aid was associated with lower standardization and no effects over time on radiologists' workload or stress. Expectations of AI decreasing the workload of radiologists were not confirmed by our real-world study.

PRE-REGISTRATION

German register for clinical trials https://drks.de/; DRKS00027391.

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

BACKGROUND

 Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.

OBJECTIVES

 In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.

METHODS

 We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.

RESULTS

 Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.

CONCLUSION

 We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

Post-implementation usability evaluation of a human factors-based clinical decision support for pulmonary embolism (PE) diagnosis (Dx): PE Dx Study Part 1

While there is promise for health IT, such as Clinical Decision Support (CDS), to improve patient safety and clinician efficiency, poor usability has hindered widespread use of these tools. Human Factors (HF) principles and methods remain the gold standard for health IT design; however, there is limited information on how HF methods and principles influence CDS usability "in the wild". In this study, we explore the usability of an HF-based CDS used in the clinical environment; the CDS was designed according to a human-centered design process, which is described in Carayon et al. (2020). In this study, we interviewed 12 emergency medicine physicians, identifying 294 excerpts of barriers and facilitators of the CDS. Sixty-eight percent of excerpts related to the HF principles applied in the human-centered design of the CDS. The remaining 32% of excerpts related to 18 inductively-created categories, which highlight gaps in the CDS design process. Several barriers were related to the physical environment and organization work system elements as well as physicians' broader workflow in the emergency department (e.g., teamwork). This study expands our understanding of the usability outcomes of HF-based CDS "in the wild". We demonstrate the value of HF principles in the usability of CDS and identify areas for improvement to future human-centered design of CDS. The relationship between these usability outcomes and the HCD process is explored in an accompanying Part 2 manuscript.

Retrospective analysis of the human-centered design process used to develop a clinical decision support in the emergency department: PE Dx Study Part 2

With the growing implementation and use of health IT such as Clinical Decision Support (CDS), there is increasing attention on the potential negative impact of these technologies on patients (e.g., medication errors) and clinicians (e.g., increased workload, decreased job satisfaction, burnout). Human-Centered Design (HCD) and Human Factors (HF) principles are recommended to improve the usability of health IT and reduce its negative impact on patients and clinicians; however, challenges persist. The objective of this study is to understand how an HCD process influences the usability of health IT. We conducted a systematic retrospective analysis of the HCD process used in the design of a CDS for pulmonary embolism diagnosis in the emergency department (ED). Guided by the usability outcomes (e.g., barriers and facilitators) of the CDS use "in the wild" (see Part 1 of this research in the accompanying manuscript), we performed deductive content analysis of 17 documents (e.g., design session transcripts) produced during the HCD process. We describe if and how the design team considered the barriers and facilitators during the HCD process. We identified 7 design outcomes of the HCD process, for instance designing a workaround and making a design change to the CDS. We identify gaps in the current HCD process and demonstrate the need for a continuous health IT design process.

The decision aid is the easy part: workflow challenges of shared decision making in cancer care

Delivering high-quality, patient-centered cancer care remains a challenge. Both the National Academy of Medicine and the American Society of Clinical Oncology recommend shared decision making to improve patient-centered care, but widespread adoption of shared decision making into clinical care has been limited. Shared decision making is a process in which a patient and the patient's health-care professional weigh the risks and benefits of different options and come to a joint decision on the best course of action for that patient on the basis of their values, preferences, and goals for care. Patients who engage in shared decision making report higher quality of care, whereas patients who are less involved in these decisions have statistically significantly higher decisional regret and are less satisfied. Decision aids can improve shared decision making-for example, by eliciting patient values and preferences that can then be shared with clinicians and by providing patients with information that may influence their decisions. However, integrating decision aids into the workflows of routine care is challenging. In this commentary, we explore 3 workflow-related barriers to shared decision making: the who, when, and how of decision aid implementation in clinical practice. We introduce readers to human factors engineering and demonstrate its potential value to decision aid design through a case study of breast cancer surgical treatment decision making. By better employing the methods and principles of human factors engineering, we can improve decision aid integration, shared decision making, and ultimately patient-centered cancer outcomes.

Physicians' Perceptions of Clinical Decision Support to Treat Patients With Heart Failure in the ED

Importance

Clinical decision support (CDS) could help emergency department (ED) physicians treat patients with heart failure (HF) by estimating risk, collating relevant history, and assisting with medication prescribing if physicians' perspectives inform its design and implementation.

Objective

To evaluate CDS usability and workflow integration in the hands of ED physician end users who use it in clinical practice.

Design, Setting, and Participants

This mixed-methods qualitative study administered semistructured interviews to ED physicians from 2 community EDs of Kaiser Permanente Northern California in 2023. The interview guide, based on the Usability Heuristics for User Interface Design and the Sociotechnical Environment models, yielded themes used to construct an electronic survey instrument sent to all ED physicians.

Main Outcomes and Measures

Main outcomes were physicians' perceptions of using CDS to complement clinical decision-making, usability, and integration into ED clinical workflow.

Results

Seven key informant physicians (5 [71.4%] female, median [IQR] 15.0 [9.5-15.0] years in practice) were interviewed and survey responses from 51 physicians (23 [45.1%] female, median [IQR] 14.0 [9.5-17.0] years in practice) were received from EDs piloting the CDS intervention. Response rate was 67.1% (51 of 76). Physicians suggested changes to CDS accessibility, functionality, and workflow integration. Most agreed that CDS would improve patient care and fewer than half of physicians expressed hesitation about their capacity to consistently comply with its recommendations, citing workload concerns. Physicians preferred a passive prompt that encouraged, but did not mandate, interaction with the CDS.

Conclusions and Relevance

In this qualitative study of physicians who were using a novel CDS intervention to assist with ED management of patients with acute HF, several opportunities were identified to improve usability as well as several key barriers and facilitators to CDS implementation.

Asynchronous Conferencing Through a Secure Messaging Application Increases Reporting of Medical Errors in a Mature Trauma Center

Background

Medical errors occur frequently, yet they are often under-reported and strategies to increase the reporting of medical errors are lacking. In this work, we detail how a level 1 trauma center used a secure messaging application to track medical errors and enhance its quality improvement initiatives.

Methods

We describe the formulation, implementation, evolution, and evaluation of a chatroom integrated into a secure texting system to identify performance improvement and patient safety (PIPS) concerns. For evaluation, we used descriptive statistics to examine PIPS reporting by the reporting method over time, the incidence of mortality and unplanned ICU readmissions tracked in the hospital trauma registry over the same, and time-to-loop closure over the study period to quantify the impact of the processes instituted by the PIPS team. We also categorized themes of reported events.

Results

With the implementation of a PIPS chatroom, the number of events reported each month increased and texting became the predominant way for users to report trauma PIPS events. This increase in PIPS reporting did not appear to be accompanied by an increase in mortality and unplanned ICU readmissions. The PIPS team also improved the tracking and timely resolution of PIPS events and observed a decrease in time-to-loop closure with the implementation of the PIPS chatroom.

Conclusions

The adoption of clinical texting as a way to report PIPS events was associated with increased reporting of such events and more timely resolution of concerns regarding patient safety and healthcare quality.

Effectiveness of an Emergency Department-Based Machine Learning Clinical Decision Support Tool to Prevent Outpatient Falls Among Older Adults: Protocol for a Quasi-Experimental Study

BACKGROUND

Emergency department (ED) providers are important collaborators in preventing falls for older adults because they are often the first health care providers to see a patient after a fall and because at-home falls are often preceded by previous ED visits. Previous work has shown that ED referrals to falls interventions can reduce the risk of an at-home fall by 38%. Screening patients at risk for a fall can be time-consuming and difficult to implement in the ED setting. Machine learning (ML) and clinical decision support (CDS) offer the potential of automating the screening process. However, it remains unclear whether automation of screening and referrals can reduce the risk of future falls among older patients.

OBJECTIVE

The goal of this paper is to describe a research protocol for evaluating the effectiveness of an automated screening and referral intervention. These findings will inform ongoing discussions about the use of ML and artificial intelligence to augment medical decision-making.

METHODS

To assess the effectiveness of our program for patients receiving the falls risk intervention, our primary analysis will be to obtain referral completion rates at 3 different EDs. We will use a quasi-experimental design known as a sharp regression discontinuity with regard to intent-to-treat, since the intervention is administered to patients whose risk score falls above a threshold. A conditional logistic regression model will be built to describe 6-month fall risk at each site as a function of the intervention, patient demographics, and risk score. The odds ratio of a return visit for a fall and the 95% CI will be estimated by comparing those identified as high risk by the ML-based CDS (ML-CDS) and those who were not but had a similar risk profile.

RESULTS

The ML-CDS tool under study has been implemented at 2 of the 3 EDs in our study. As of April 2023, a total of 1326 patient encounters have been flagged for providers, and 339 unique patients have been referred to the mobility and falls clinic. To date, 15% (45/339) of patients have scheduled an appointment with the clinic.

CONCLUSIONS

This study seeks to quantify the impact of an ML-CDS intervention on patient behavior and outcomes. Our end-to-end data set allows for a more meaningful analysis of patient outcomes than other studies focused on interim outcomes, and our multisite implementation plan will demonstrate applicability to a broad population and the possibility to adapt the intervention to other EDs and achieve similar results. Our statistical methodology, regression discontinuity design, allows for causal inference from observational data and a staggered implementation strategy allows for the identification of secular trends that could affect causal associations and allow mitigation as necessary.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05810064; https://www.clinicaltrials.gov/study/NCT05810064.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/48128.

Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review

BACKGROUND

Successful implementation and utilization of Computerized Clinical Decision Support Systems (CDSS) in hospitals is complex and challenging. Implementation science, and in particular the Nonadoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework, may offer a systematic approach for identifying and addressing these challenges. This review aimed to identify, categorize, and describe barriers and facilitators to CDSS implementation in hospital settings and map them to the NASSS framework. Exploring the applicability of the NASSS framework to CDSS implementation was a secondary aim.

METHODS

Electronic database searches were conducted (21 July 2020; updated 5 April 2022) in Ovid MEDLINE, Embase, Scopus, PyscInfo, and CINAHL. Original research studies reporting on measured or perceived barriers and/or facilitators to implementation and adoption of CDSS in hospital settings, or attitudes of healthcare professionals towards CDSS were included. Articles with a primary focus on CDSS development were excluded. No language or date restrictions were applied. We used qualitative content analysis to identify determinants and organize them into higher-order themes, which were then reflexively mapped to the NASSS framework.

RESULTS

Forty-four publications were included. These comprised a range of study designs, geographic locations, participants, technology types, CDSS functions, and clinical contexts of implementation. A total of 227 individual barriers and 130 individual facilitators were identified across the included studies. The most commonly reported influences on implementation were fit of CDSS with workflows (19 studies), the usefulness of the CDSS output in practice (17 studies), CDSS technical dependencies and design (16 studies), trust of users in the CDSS input data and evidence base (15 studies), and the contextual fit of the CDSS with the user's role or clinical setting (14 studies). Most determinants could be appropriately categorized into domains of the NASSS framework with barriers and facilitators in the "Technology," "Organization," and "Adopters" domains most frequently reported. No determinants were assigned to the "Embedding and Adaptation Over Time" domain.

CONCLUSIONS

This review identified the most common determinants which could be targeted for modification to either remove barriers or facilitate the adoption and use of CDSS within hospitals. Greater adoption of implementation theory should be encouraged to support CDSS implementation.

Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework

Alerts are a common functionality of clinical decision support systems (CDSSs). Although they have proven to be useful in clinical practice, the alert burden can lead to alert fatigue and significantly reduce their usability and acceptance. Based on a literature review, we propose a unified framework consisting of a set of meaningful timestamps that allows the use of state-of-the-art measures for alert burden, such as alert dwell time, alert think time, and response time. In addition, it can be used to investigate other measures that could be relevant as regards dealing with this problem. Furthermore, we provide a case study concerning three different types of alerts to which the framework was successfully applied. We consider that our framework can easily be adapted to other CDSSs and that it could be useful for dealing with alert burden measurement thus contributing to its appropriate management.

Adoption of shared decision-making and clinical decision support for reducing cardiovascular disease risk in community health centers

Objective

Electronic health record (EHR)-based shared decision-making (SDM) and clinical decision support (CDS) systems can improve cardiovascular disease (CVD) care quality and risk factor management. Use of the CV Wizard system showed a beneficial effect on high-risk community health center (CHC) patients' CVD risk within an effectiveness trial, but system adoption was low overall. We assessed which multi-level characteristics were associated with system use.

Materials and Methods

Analyses included 80 195 encounters with 17 931 patients with high CVD risk and/or uncontrolled risk factors at 42 clinics in September 2018-March 2020. Data came from the CV Wizard repository and EHR data, and a survey of 44 clinic providers. Adjusted, mixed-effects multivariate Poisson regression analyses assessed factors associated with system use. We included clinic- and provider-level clustering as random effects to account for nested data.

Results

Likelihood of system use was significantly higher in encounters with patients with higher CVD risk and at longer encounters, and lower when providers were >10 minutes behind schedule, among other factors. Survey participants reported generally high satisfaction with the system but were less likely to use it when there were time constraints or when rooming staff did not print the system output for the provider.

Discussion

CHC providers prioritize using this system for patients with the greatest CVD risk, when time permits, and when rooming staff make the information readily available. CHCs' financial constraints create substantial challenges to addressing barriers to improved system use, with health equity implications.

Conclusion

Research is needed on improving SDM and CDS adoption in CHCs.

Trial Registration

ClinicalTrials.gov, NCT03001713, https://clinicaltrials.gov/.

A Sociotechnical Systems Framework for the Application of Artificial Intelligence in Health Care Delivery

In the coming years, artificial intelligence (AI) will pervade almost every aspect of the health care delivery system. AI has the potential to improve patient safety (e.g. diagnostic accuracy) as well as reduce the burden on clinicians (e.g. documentation-related workload); however, these benefits are yet to be realized. AI is only one element of a larger sociotechnical system that needs to be considered for effective AI application. In this paper, we describe the current challenges of integrating AI into clinical care and propose a sociotechnical systems (STS) approach for AI design and implementation. We demonstrate the importance of an STS approach through a case study on the design and implementation of a clinical decision support (CDS). In order for AI to reach its potential, the entire work system as well as clinical workflow must be systematically considered throughout the design of AI technology.

Clinician perceptions of a clinical decision support system to reduce cardiovascular risk among prediabetes patients in a predominantly rural healthcare system

Background

The early detection and management of uncontrolled cardiovascular risk factors among prediabetes patients can prevent cardiovascular disease (CVD). Prediabetes increases the risk of CVD, which is a leading cause of death in the United States. CVD clinical decision support (CDS) in primary care settings has the potential to reduce cardiovascular risk in patients with prediabetes while potentially saving clinicians time. The objective of this study is to understand primary care clinician (PCC) perceptions of a CDS system designed to reduce CVD risk in adults with prediabetes.

Methods

We administered pre-CDS implementation (6/30/2016 to 8/25/2016) (n = 183, 61% response rate) and post-CDS implementation (6/12/2019 to 8/7/2019) (n = 131, 44.5% response rate) independent cross-sectional electronic surveys to PCCs at 36 randomized primary care clinics participating in a federally funded study of a CVD risk reduction CDS tool. Surveys assessed PCC demographics, experiences in delivering prediabetes care, perceptions of CDS impact on shared decision making, perception of CDS impact on control of major CVD risk factors, and overall perceptions of the CDS tool when managing cardiovascular risk.

Results

We found few significant differences when comparing pre- and post-implementation responses across CDS intervention and usual care (UC) clinics. A majority of PCCs felt well-prepared to discuss CVD risk factor control with patients both pre- and post-implementation. About 73% of PCCs at CDS intervention clinics agreed that the CDS helped improve risk control, 68% reported the CDS added value to patient clinic visits, and 72% reported they would recommend use of this CDS system to colleagues. However, most PCCs disagreed that the CDS saves time talking about preventing diabetes or CVD, and most PCCs also did not find the clinical domains useful, nor did PCCs believe that the clinical domains were useful in getting patients to take action. Finally, only about 38% reported they were satisfied with the CDS.

Conclusions

These results improve our understanding of CDS user experience and can be used to guide iterative improvement of the CDS. While most PCCs agreed the CDS improves CVD and diabetes risk factor control, they were generally not satisfied with the CDS. Moreover, only 40–50% agreed that specific suggestions on clinical domains helped patients to take action. In spite of this, an overwhelming majority reported they would recommend the CDS to colleagues, pointing for the need to improve upon the current CDS.

Trial registration: NCT02759055 03/05/2016.

Barriers and Opportunities Regarding Implementation of a Machine Learning-Based Acute Heart Failure Risk Stratification Tool in the Emergency Department

Hospital admissions for patients with acute heart failure (AHF) remain high. There is an opportunity to improve alignment between patient risk and admission decision. We recently developed a machine learning (ML)-based model that stratifies emergency department (ED) patients with AHF based on predicted risk of a 30-day severe adverse event. Prior to deploying the algorithm and paired clinical decision support, we sought to understand barriers and opportunities regarding successful implementation. We conducted semi-structured interviews with eight front-line ED providers and surveyed 67 ED providers. Audio-recorded interviews were transcribed and analyzed using thematic analysis, and we had a 65% response rate to the survey. Providers wanted decision support to be streamlined into workflows with minimal disruptions. Most providers wanted assistance primarily with ED disposition decisions, and secondarily with medical management and post-discharge follow-up care. Receiving feedback on patient outcomes after risk tool use was seen as an opportunity to increase acceptance, and few providers (<10%) had significant hesitations with using an ML-based tool after education on its use. Engagement with key front-line users on optimal design of the algorithm and decision support may contribute to broader uptake, acceptance, and adoption of recommendations for clinical decisions.

Human factors applications in the design of decision support systems for population health: a scoping review

INTRODUCTION

Public health professionals engage in complex cognitive tasks, often using evidence-based decision support tools to bolster their decision-making. Human factors methods take a user-centred approach to improve the design of systems, processes, and interfaces to better support planning and decision-making. While human factors methods have been applied to the design of clinical health tools, these methods are limited in the design of tools for population health. The objective of this scoping review is to develop a comprehensive understanding of how human factors techniques have been applied in the design of population health decision support tools.

METHODS AND ANALYSIS

The scoping review will follow the methodology and framework proposed by Arksey and O'Malley. We include English-language documents between January 1990 and August 2021 describing the development, validation or application of human factors principles to decision support tools in population health. The search will include Ovid MEDLINE: Epub Ahead of Print, In-Process and Other Non-Indexed Citations, Ovid MEDLINE Daily and Ovid MEDLINE 1946-present; EMBASE, Scopus, PsycINFO, Compendex, IEEE Xplore and Inspec. The results will be integrated into Covidence. First, the abstract of all identified articles will be screened independently by two reviewers with disagreements being resolved by a third reviewer. Next, the full text for articles identified as include or inconclusive will be reviewed by two independent reviewers, leading to a final decision regarding inclusion. Reference lists of included articles will be manually screened to identify additional studies. Data will be extracted by one reviewer, verified by a second, and presented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews.

ETHICS AND DISSEMINATION

Ethics approval is not required for this work as human participants are not involved. The completed review will be published in a peer-reviewed, interdisciplinary journal.

Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments

Of the 3 million older adults seeking fall-related emergency care each year, nearly one-third visited the Emergency Department (ED) in the previous 6 months. ED providers have a great opportunity to refer patients for fall prevention services at these initial visits, but lack feasible tools for identifying those at highest-risk. Existing fall screening tools have been poorly adopted due to ED staff/provider burden and lack of workflow integration. To address this, we developed an automated clinical decision support (CDS) system for identifying and referring older adult ED patients at risk of future falls. We engaged an interdisciplinary design team (ED providers, health services researchers, information technology/predictive analytics professionals, and outpatient Falls Clinic staff) to collaboratively develop a system that successfully met user requirements and integrated seamlessly into existing ED workflows. Our rapid-cycle development and evaluation process employed a novel combination of human-centered design, implementation science, and patient experience strategies, facilitating simultaneous design of the CDS tool and intervention implementation strategies. This included defining system requirements, systematically identifying and resolving usability problems, assessing barriers and facilitators to implementation (e.g., data accessibility, lack of time, high patient volumes, appointment availability) from multiple vantage points, and refining protocols for communicating with referred patients at discharge. ED physician, nurse, and patient stakeholders were also engaged through online surveys and user testing. Successful CDS design and implementation required integration of multiple new technologies and processes into existing workflows, necessitating interdisciplinary collaboration from the onset. By using this iterative approach, we were able to design and implement an intervention meeting all project goals. Processes used in this Clinical-IT-Research partnership can be applied to other use cases involving automated risk-stratification, CDS development, and EHR-facilitated care coordination.

Usability barriers and facilitators of a human factors engineering-based clinical decision support technology for diagnosing pulmonary embolism

BACKGROUND

Health IT, such as clinical decision support (CDS), has the potential to improve patient safety. However, poor usability of health IT continues to be a major concern. Human factors engineering (HFE) approaches are recommended to improve the usability of health IT. Limited evidence exists on the actual impact of HFE methods and principles on the usability of health IT.

OBJECTIVE

To identify and describe the usability barriers and facilitators of an HFE-based CDS prior to implementation in the emergency department (ED).

METHODS

We conducted debrief interviews with 32 emergency medicine physicians as a part of a scenario-based simulation study evaluating the usability of the HFE-based CDS. We performed a deductive content analysis of the interviews using the usability criteria of Scapin and Bastien as a framework.

RESULTS

We identified 271 occurrences of usability barriers (94) and facilitators (177) of the HFE-based CDS. For instance, we found a facilitator relating to the usability criteria prompting as the PE Dx helps the physician order diagnostic tests following the risk assessment. We found the most facilitators relating to the criteria, minimal actions, e.g. as the PE Dx automatically populates vitals signs (e.g., heart rate) from the chart into the CDS. The majority of the usability barriers related to the usability criteria, compatibility (i.e., workflow integration), which was not explicitly considered in the HFE design of the CDS. For example, the CDS did not support resident and attending physician teamwork in the PE diagnostic process.

CONCLUSION

The systematic use of HFE principles in the design of CDS improves the usability of these technologies. In order to further reduce usability barriers, workflow integration should be explicitly considered in the design of health IT.