Analysis of clinical decision support system malfunctions: a case series and survey

Extubation Advisor: Implementation and Evaluation of A Novel Extubation Clinical Decision Support Tool

IMPORTANCE

Extubation Advisor (EA) is a novel software tool that generates a synoptic report for each Spontaneous Breathing Trial (SBT) conducted to inform extubation decision-making.

OBJECTIVES

To assess bedside EA implementation, perceptions of utility, and identify barriers and facilitators of use.

DESIGN, SETTING AND PARTICIPANTS

We conducted a phase I mixed-methods interventional study in three mixed intensive care unit (ICUs) in two academic hospitals. We interviewed critical care physicians (MDs) and respiratory therapists (RTs) regarding user-centered design principles and usability.

ANALYSIS

We evaluated our ability to consent participants (feasibility threshold 50%), capture complete data (threshold 90%), generate and review EA reports in real-time (thresholds 75% and 80%, respectively), and MD perception of tool usefulness (6-point Likert scale). We analyzed interview transcripts using inductive coding to identify facilitators and barriers to EA implementation and perceived benefit of tool use.

RESULTS

We enrolled 31 patients who underwent 70 SBTs. Although consent rates [31/31 (100%], complete data capture [68/68 (100%)], and EA report generation [68/70 (97.1%)] exceeded feasibility thresholds, reports were reviewed by MDs for [55/70 (78.6%)] SBTs. Mean MD usefulness score was 4.0/6. Based on feedback obtained from 36 interviews (15 MDs, 21 RTs), we revised the EA report twice and identified facilitators (ability to track patient progress, enhance extubation decision-making, and provide support in resource-limited settings) and barriers (resource constraints, need for education) to tool implementation. Half of respondents (9 MDs, 9 RTs; combined 50%) perceived definite or potential benefit to EA tool use.

CONCLUSION

This is the first study of a waveform-based variability-derived, predictive clinical decision support tool evaluated in adult ICUs. Our findings support the feasibility of integrating the EA tool into bedside workflow. Clinical trials are needed to assess the utility of the EA tool in practice and its impact on extubation decision-making and outcomes.

TRIAL REGISTRATION

NCT04708509.

A scoping review of rule-based clinical decision support malfunctions

OBJECTIVE

Conduct a scoping review of research studies that describe rule-based clinical decision support (CDS) malfunctions.

MATERIALS AND METHODS

In April 2022, we searched three bibliographic databases (MEDLINE, CINAHL, and Embase) for literature referencing CDS malfunctions. We coded the identified malfunctions according to an existing CDS malfunction taxonomy and added new categories for factors not already captured. We also extracted and summarized information related to the CDS system, such as architecture, data source, and data format.

RESULTS

Twenty-eight articles met inclusion criteria, capturing 130 malfunctions. Architectures used included stand-alone systems (eg, web-based calculator), integrated systems (eg, best practices alerts), and service-oriented architectures (eg, distributed systems like SMART or CDS Hooks). No standards-based CDS malfunctions were identified. The "Cause" category of the original taxonomy includes three new types (organizational policy, hardware error, and data source) and two existing causes were expanded to include additional layers. Only 29 malfunctions (22%) described the potential impact of the malfunction on patient care.

DISCUSSION

While a substantial amount of research on CDS exists, our review indicates there is a limited focus on CDS malfunctions, with even less attention on malfunctions associated with modern delivery architectures such as SMART and CDS Hooks.

CONCLUSION

CDS malfunctions can and do occur across several different care delivery architectures. To account for advances in health information technology, existing taxonomies of CDS malfunctions must be continually updated. This will be especially important for service-oriented architectures, which connect several disparate systems, and are increasing in use.

Impact of a Clinical Decision Support Alert on Informed Consent Documentation in the Neonatal Intensive Care Unit

Background

Informed consent is necessary to preserve patient autonomy and shared decision-making, yet compliant consent documentation is suboptimal in the intensive care unit (ICU). We aimed to increase compliance with bundled consent documentation, which provides consent for a predefined set of common procedures in the neonatal ICU from 0% to 50% over 1 year.

Methods

We used the Plan-Do-Study-Act model for quality improvement. Interventions included education and performance awareness, delineation of the preferred consenting process, consent form revision, overlay tool creation, and clinical decision support (CDS) alert use within the electronic health record. Monthly audits categorized consent forms as missing, present but noncompliant, or compliant. We analyzed consent compliance on a run chart using standard run chart interpretation rules and obtained feedback on the CDS as a countermeasure.

Results

We conducted 564 audits over 37 months. Overall, median consent compliance increased from 0% to 86.6%. Upon initiating the CDS alert, we observed the highest monthly compliance of 93.3%, followed by a decrease to 33.3% with an inadvertent discontinuation of the CDS. Compliance subsequently increased to 73.3% after the restoration of the alert. We created a consultant opt-out selection to address negative feedback associated with CDS. There were no missing consent forms within the last 7 months of monitoring.

Conclusions

A multi-faceted approach led to sustained improvement in bundled consent documentation compliance in our neonatal intensive care unit, with the direct contribution of the CDS observed. A CDS intervention directed at the informed consenting process may similarly benefit other ICUs.

AI and machine learning ethics, law, diversity, and global impact

Artificial intelligence (AI) and its machine learning (ML) algorithms are offering new promise for personalized biomedicine and more cost-effective healthcare with impressive technical capability to mimic human cognitive capabilities. However, widespread application of this promising technology has been limited in the medical domain and expectations have been tampered by ethical challenges and concerns regarding patient privacy, legal responsibility, trustworthiness, and fairness. To balance technical innovation with ethical applications of AI/ML, developers must demonstrate the AI functions as intended and adopt strategies to minimize the risks for failure or bias. This review describes the new ethical challenges created by AI/ML for clinical care and identifies specific considerations for its practice in medicine. We provide an overview of regulatory and legal issues applicable in Europe and the United States, a description of technical aspects to consider, and present recommendations for trustworthy AI/ML that promote transparency, minimize risks of bias or error, and protect the patient well-being.

Use of an open-source electronic health record to establish a "virtual hospital": A tale of two curricula

BACKGROUND

The electronic health record (EHR) is central to medical informatics. Its use is also recognized as an important skill for future clinicians. Typically, medical students' first exposure to an EHR is when they start their clinical internships, and medical informatics students may or may not get experience with an EHR before graduation. We describe the process of implementing an open-source EHR in two curricula: Medicine and Medical informatics. For medical students, the primary goals were to allow students to practice analyzing information from the EHR, creating therapeutic plans, and communicating with their colleagues via the EHR before they start their first clinical rotations. For medical informatics students, the primary goal was to give students hands-on experience with creating decision support in an EHR.

APPROACH

We used the OpenMRS electronic health record with a custom decision support module based on Arden Syntax. Medical students needed a secure, stable environment to practice medical reasoning. Medical informatics students needed a more isolated system to experiment with the EHR's internal configuration. Both student groups needed synthetic patient cases that were realistic, but in different aspects. For medical students, it is essential that these cases are clinically consistent, and events unfold in a logical order. By contrast, synthetic data for medical informatics students should mimic the data quality problems found in real patient data.

OUTCOMES

Medical informatics students show more mature reasoning about data quality issues and workflow integration than prior to using the EHR. Comments on both course evaluations have been positive, including comments on how working with a real-world EHR provides a realistic experience.

CONCLUSION

The open-source EHR OpenMRS has proven to be a valuable addition to both the medicine and medical informatics curriculum. Both sets of students experience use of the EHR as giving them valuable, realistic learning experiences.

Real-Time User Feedback to Support Clinical Decision Support System Improvement

OBJECTIVES

To improve clinical decision support (CDS) by allowing users to provide real-time feedback when they interact with CDS tools and by creating processes for responding to and acting on this feedback.

METHODS

Two organizations implemented similar real-time feedback tools and processes in their electronic health record and gathered data over a 30-month period. At both sites, users could provide feedback by using Likert feedback links embedded in all end-user facing alerts, with results stored outside the electronic health record, and provide feedback as a comment when they overrode an alert. Both systems are monitored daily by clinical informatics teams.

RESULTS

The two sites received 2,639 Likert feedback comments and 623,270 override comments over a 30-month period. Through four case studies, we describe our use of end-user feedback to rapidly respond to build errors, as well as identifying inaccurate knowledge management, user-interface issues, and unique workflows.

CONCLUSION

Feedback on CDS tools can be solicited in multiple ways, and it contains valuable and actionable suggestions to improve CDS alerts. Additionally, end users appreciate knowing their feedback is being received and may also make other suggestions to improve the electronic health record. Incorporation of end-user feedback into CDS monitoring, evaluation, and remediation is a way to improve CDS.

Leveraging law and ethics to promote safe and reliable AI/ML in healthcare

Artificial intelligence and machine learning (AI/ML) is poised to disrupt the structure and delivery of healthcare, promising to optimize care clinical care delivery and information management. AI/ML offers potential benefits in healthcare, such as creating novel clinical decision support tools, pattern recognition software, and predictive modeling systems. This raises questions about how AI/ML will impact the physician-patient relationship and the practice of medicine. Effective utilization and reliance on AI/ML also requires that these technologies are safe and reliable. Potential errors could not only pose serious risks to patient safety, but also expose physicians, hospitals, and AI/ML manufacturers to liability. This review describes how the law provides a mechanism to promote safety and reliability of AI/ML systems. On the front end, the Food and Drug Administration (FDA) intends to regulate many AI/ML as medical devices, which corresponds to a set of regulatory requirements prior to product marketing and use. Post-development, a variety of mechanisms in the law provide guardrails for careful deployment into clinical practice that can also incentivize product improvement. This review provides an overview of potential areas of liability arising from AI/ML including malpractice, informed consent, corporate liability, and products liability. Finally, this review summarizes strategies to minimize risk and promote safe and reliable AI/ML.

Clinical decision support malfunctions related to medication routes: a case series

Objective

To identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them.

Materials and Methods

Case series of medication route-related CDS malfunctions from diverse healthcare provider organizations.

Results

Nine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate.

Discussion

These value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management.

Conclusion

Value set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.

Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing

Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments at early time points, which is critical for improving sepsis outcomes. In view of the increasing use of such systems, better understanding of how they are adopted and used by healthcare providers is needed. Here, we analyzed provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System), which was deployed at five hospitals over a 2-year period. Among 9,805 retrospectively identified sepsis cases, the early detection tool achieved high sensitivity (82% of sepsis cases were identified) and a high rate of adoption: 89% of all alerts by the system were evaluated by a physician or advanced practice provider and 38% of evaluated alerts were confirmed by a provider. Adjusting for patient presentation and severity, patients with sepsis whose alert was confirmed by a provider within 3 h had a 1.85-h (95% CI 1.66-2.00) reduction in median time to first antibiotic order compared to patients with sepsis whose alert was either dismissed, confirmed more than 3 h after the alert or never addressed in the system. Finally, we found that emergency department providers and providers who had previous interactions with an alert were more likely to interact with alerts, as well as to confirm alerts on retrospectively identified patients with sepsis. Beyond efforts to improve the performance of early warning systems, efforts to improve adoption are essential to their clinical impact and should focus on understanding providers' knowledge of, experience with and attitudes toward such systems.

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Clinician collaboration to improve clinical decision support: the Clickbusters initiative

OBJECTIVE

We describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts.

MATERIALS AND METHODS

We developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program.

RESULTS

Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.

CONCLUSIONS

At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.

Alerts in Clinical Decision Support Systems (CDSS): A Bibliometric Review and Content Analysis

A clinical decision support system (CDSS) informs or generates medical recommendations for healthcare practitioners. An alert is the most common way for a CDSS to interact with practitioners. Research about alerts in CDSS has proliferated over the past ten years. The research trend is ongoing with new emerging terms and focus. Bibliometric analysis is ideal for researchers to understand the research trend and future directions. Influential articles, institutes, countries, authors, and commonly used keywords were analyzed to grasp a comprehensive view on our topic, alerts in CDSS. Articles published between 2011 and 2021 were extracted from the Web of Science database. There were 728 articles included for bibliometric analysis, among which 24 papers were selected for content analysis. Our analysis shows that the research direction has shifted from patient safety to system utility, implying the importance of alert usability to be clinically impactful. Finally, we conclude with future research directions such as the optimization of alert mechanisms and comprehensiveness to enhance alert appropriateness and to reduce alert fatigue.

Functional IT Complementarity and Hospital Performance in the United States: A Longitudinal Investigation

For healthcare practitioners and policymakers, one of the most challenging problems is understanding how to implement health information technology (HIT) applications in a way that yields the most positive impacts on quality and cost of care. We identify four clinical HIT functions which we label as order entry and management (OEM), decision support (DS), electronic clinical documentation (ECD), and results viewing (RV). We view OEM and DS as primary clinical functions and ECD and RV as support clinical functions. Our results show that no single combination of applications uniformly improves clinical and experiential quality and reduces cost for all hospitals. Thus, managers must assess which HIT interactions improve which performance metric under which conditions. Our results suggest that synergies can be realized when these systems are implemented simultaneously. Additionally, synergies can occur when support HIT is implemented before primary HIT and irrespective of the order in which primary HITs are implemented. Practitioners should also be aware that the synergistic effects of HITs and their impact on cost and quality are different for chronic and acute diseases. Our key message to top managers is to prioritize different combinations of HIT contingent on the performance variables they are targeting for their hospitals but also to realize that technology may not impact all outcomes.

Acceptance, barriers and facilitators to implementing AI-based decision support systems in emergency departments: a quantitative and qualitative evaluation (Preprint)

Background

Despite the increasing availability of clinical decision support systems (CDSSs) and rising expectation for CDSSs based on artificial intelligence (AI), little is known about the acceptance of AI-based CDSS by physicians and its barriers and facilitators in emergency care settings.

Objective

We aimed to evaluate the acceptance, barriers, and facilitators to implementing AI-based CDSSs in the emergency care setting through the opinions of physicians on our newly developed, real-time AI-based CDSS, which alerts ED physicians by predicting aortic dissection based on numeric and text information from medical charts, by using the Unified Theory of Acceptance and Use of Technology (UTAUT; for quantitative evaluation) and the Consolidated Framework for Implementation Research (CFIR; for qualitative evaluation) frameworks.

Methods

This mixed methods study was performed from March to April 2021. Transitional year residents (n=6), emergency medicine residents (n=5), and emergency physicians (n=3) from two community, tertiary care hospitals in Japan were included. We first developed a real-time CDSS for predicting aortic dissection based on numeric and text information from medical charts (eg, chief complaints, medical history, vital signs) with natural language processing. This system was deployed on the internet, and the participants used the system with clinical vignettes of model cases. Participants were then involved in a mixed methods evaluation consisting of a UTAUT-based questionnaire with a 5-point Likert scale (quantitative) and a CFIR-based semistructured interview (qualitative). Cronbach α was calculated as a reliability estimate for UTAUT subconstructs. Interviews were sampled, transcribed, and analyzed using the MaxQDA software. The framework analysis approach was used during the study to determine the relevance of the CFIR constructs.

Results

All 14 participants completed the questionnaires and interviews. Quantitative analysis revealed generally positive responses for user acceptance with all scores above the neutral score of 3.0. In addition, the mixed methods analysis identified two significant barriers (System Performance, Compatibility) and two major facilitators (Evidence Strength, Design Quality) for implementation of AI-based CDSSs in emergency care settings.

Conclusions

Our mixed methods evaluation based on theoretically grounded frameworks revealed the acceptance, barriers, and facilitators of implementation of AI-based CDSS. Although the concern of system failure and overtrusting of the system could be barriers to implementation, the locality of the system and designing an intuitive user interface could likely facilitate the use of optimal AI-based CDSS. Alleviating and resolving these factors should be key to achieving good user acceptance of AI-based CDSS.

Clinical Decision Support for Laboratory Testing

BACKGROUND

As technology enables new and increasingly complex laboratory tests, test utilization presents a growing challenge for healthcare systems. Clinical decision support (CDS) refers to digital tools that present providers with clinically relevant information and recommendations, which have been shown to improve test utilization. Nevertheless, individual CDS applications often fail, and implementation remains challenging.

CONTENT

We review common classes of CDS tools grounded in examples from the literature as well as our own institutional experience. In addition, we present a practical framework and specific recommendations for effective CDS implementation.

SUMMARY

CDS encompasses a rich set of tools that have the potential to drive significant improvements in laboratory testing, especially with respect to test utilization. Deploying CDS effectively requires thoughtful design and careful maintenance, and structured processes focused on quality improvement and change management play an important role in achieving these goals.

Feasibility of implementing Extubation Advisor, a clinical decision support tool to improve extubation decision-making in the ICU: a mixed-methods observational study

OBJECTIVES

Although spontaneous breathing trials (SBTs) are standard of care to extubation readiness, no tool exists that optimises prediction and standardises assessment. In this study, we evaluated the feasibility and clinical impressions of Extubation Advisor (EA), a comprehensive clinical extubation decision support (CDS) tool.

DESIGN

Phase I mixed-methods observational study.

SETTING

Two Canadian intensive care units (ICUs).

PARTICIPANTS

We included patients on mechanical ventilation for ≥24 hours and clinicians (respiratory therapists and intensivists) responsible for extubation decisions.

INTERVENTIONS

Components included a predictive model assessment, feasibility evaluation, questionnaires and interviews with clinicians.

RESULTS

We enrolled 117 patients, totalling 151 SBTs and 80 extubations. The incidence of extubation failure was 11% in low-risk patients and 21% in high-risk patients stratified by the predictive model; 38% failed extubation when both the model and clinical impression were at high risk. The tool was well rated: 94% and 75% rated the data entry and EA report as average or better, respectively. Interviews (n=15) revealed favourable impressions regarding its user interface and functionality, but unexpectedly, also concerns regarding EA's potential impact on respiratory therapists' job security.

CONCLUSIONS

EA implementation was feasible, and users perceived it to have potential to support extubation decision-making. This study helps to understand bedside implementation of CDS tools in a multidisciplinary ICU.

TRIAL REGISTRATION NUMBER

NCT02988167.

The appropriateness of clinical decision support systems alerts in supporting clinical workflows: A systematic review

A CDSS generates a high number of inappropriate alerts that interrupt the clinical workflow. As a result, clinicians silence, disable, or ignore alerts, thereby undermining patient safety. Therefore, the effectiveness and appropriateness of CDSS alerts need to be evaluated. A systematic review was carried out to identify the factors that affect CDSS alert appropriateness in supporting clinical workflow. Seven electronic databases (PubMed, Scopus, ACM, Science Direct, IEEE, Ovid Medline, and Ebscohost) were searched for English language articles published between 1997 and 2018. Seventy six papers met the inclusion criteria, of which 26, 24, 15, and 11 papers are retrospective cohort, qualitative, quantitative, and mixed-method studies, respectively. The review highlights various factors influencing the appropriateness and efficiencies of CDSS alerts. These factors are categorized into technology, human, organization, and process aspects using a combination of approaches, including socio-technical framework, five rights of CDSS, and Lean. Most CDSS alerts were not properly designed based on human factor methods and principles, explaining high alert overrides in clinical practices. The identified factors and recommendations from the review may offer valuable insights into how CDSS alerts can be designed appropriately to support clinical workflow.

[Artificial intelligence in medicine and gynecology-the wrong track or promise of cure?]

Artificial intelligence (AI) has attained a new level of maturity in recent years and is becoming the driver of digitalization in all areas of life. AI is a cross-sectional technology with great importance for all areas of medicine employing image data, text data and bio-data. There is no medical field that will remain unaffected by AI, with AI-assisted clinical decision-making assuming a particularly important role. AI methods are becoming established in medical workflow management and for prediction of treatment success or treatment outcome. AI systems are already able to lend support to imaging-based diagnosis and patient management, but cannot suggest critical decisions. The corresponding preventive or therapeutic measures can be more rationally assessed with the help of AI, although the number of diseases covered is currently too low to create robust systems for routine clinical use. Prerequisite for the widespread use of AI systems is appropriate training to enable physicians to decide when computer-assisted decision-making can be relied upon.

Electronic health record usability and workload changes over time for provider and nursing staff following transition to new EHR

The ubiquity of EHRs in healthcare means that small EHR inefficiencies can have a major impact on clinician workload. We conducted a sequential explanatory mixed methods study to: 1) identify EHR-associated workload and usability effects for clinicians following an EHR change over time, 2) determine workload and usability differences for providers (MD and Advance Practice Nurses) versus nurses (RNs and MAs), 3) determine if usability predicts workload, 4) identify potential sources of EHR design flaws. Workload (NASA-Task Load Index) and usability (System Usability Scale) measures were administered pre, 6-8 month and 30-32 months post-implementation. We found significant increase in perceived workload post-implementation that persisted for 2.5 years (p < .001). The workload increase was associated with usability ratings, which in turn may relate to EHR interface design violations identified by a heuristic evaluation. Our findings suggest further innovation and attention to interface design flaws are needed to improve EHR usability and reduce clinician workload.

Association of Electronic Health Record Vendors With Hospital Financial and Quality Performance: Retrospective Data Analysis

Background

Electronic health records (EHRs) are a central feature of care delivery in acute care hospitals; however, the financial and quality outcomes associated with system performance remain unclear.

Objective

In this study, we aimed to evaluate the association between the top 3 EHR vendors and measures of hospital financial and quality performance.

Methods

This study evaluated 2667 hospitals with Cerner, Epic, or Meditech as their primary EHR and considered their performance with regard to net income, Hospital Value–Based Purchasing Total Performance Score (TPS), and the unweighted subdomains of efficiency and cost reduction; clinical care; patient- and caregiver-centered experience; and patient safety. We hypothesized that there would be a difference among the 3 vendors for each measure.

Results

None of the EHR systems were associated with a statistically significant financial relationship in our study. Epic was positively associated with TPS outcomes (R²=23.6%; β=.0159, SE 0.0079; P=.04) and higher patient perceptions of quality (R²=29.3%; β=.0292, SE 0.0099; P=.003) but was negatively associated with patient safety quality scores (R²=24.3%; β=−.0221, SE 0.0102; P=.03). Cerner and Epic were positively associated with improved efficiency (R²=31.9%; Cerner: β=.0330, SE 0.0135, P=.01; Epic: β=.0465, SE 0.0133, P<.001). Finally, all 3 vendors were associated with positive performance in the clinical care domain (Epic: β=.0388, SE 0.0122, P=.002; Cerner: β=.0283, SE 0.0124, P=.02; Meditech: β=.0273, SE 0.0123, P=.03) but with low explanatory power (R²=4.2%).

Conclusions

The results of this study provide evidence of a difference in clinical outcome performance among the top 3 EHR vendors and may serve as supportive evidence for health care leaders to target future capital investments to improve health care delivery.

An Interdisciplinary Approach to Reducing Errors in Extracted Electronic Health Record Data for Research

Erroneous electronic health record (EHR) data capture is a barrier to preserving data integrity. We assessed the impact of an interdisciplinary process in minimizing EHR data loss from prescription orders. We implemented a three-step approach to reduce data loss due to missing medication doses: Step 1-A data analyst updated the request code to optimize data capture; Step 2-A pharmacist and physician identified variations in EHR prescription workflows; and Step 3-The clinician team determined daily doses for patients with multiple prescriptions in the same encounter. The initial report contained 1421 prescriptions, with 377 (26.5 percent) missing dosages. Missing dosages reduced to 361 (26.3 percent) prescriptions following Step 1, and twenty-three (1.7 percent) records after Step 2. After Step 3, 1210 prescriptions remained, including 16 (1.3 percent) prescriptions missing doses. Prescription data is susceptible to missing values due to multiple data capture workflows. Our approach minimized data loss, improving its validity in retrospective research.

Algorithmic Detection of Boolean Logic Errors in Clinical Decision Support Statements

OBJECTIVE

Clinical decision support (CDS) can contribute to quality and safety. Prior work has shown that errors in CDS systems are common and can lead to unintended consequences. Many CDS systems use Boolean logic, which can be difficult for CDS analysts to specify accurately. We set out to determine the prevalence of certain types of Boolean logic errors in CDS statements.

METHODS

Nine health care organizations extracted Boolean logic statements from their Epic electronic health record (EHR). We developed an open-source software tool, which implemented the Espresso logic minimization algorithm, to identify three classes of logic errors.

RESULTS

Participating organizations submitted 260,698 logic statements, of which 44,890 were minimized by Espresso. We found errors in 209 of them. Every participating organization had at least two errors, and all organizations reported that they would act on the feedback.

DISCUSSION

An automated algorithm can readily detect specific categories of Boolean CDS logic errors. These errors represent a minority of CDS errors, but very likely require correction to avoid patient safety issues. This process found only a few errors at each site, but the problem appears to be widespread, affecting all participating organizations.

CONCLUSION

Both CDS implementers and EHR vendors should consider implementing similar algorithms as part of the CDS authoring process to reduce the number of errors in their CDS interventions.

User stories as lightweight requirements for agile clinical decision support development

Objective

We sought to demonstrate applicability of user stories, progressively elaborated by testable acceptance criteria, as lightweight requirements for agile development of clinical decision support (CDS).

Materials and Methods

User stories employed the template: As a [type of user], I want [some goal] so that [some reason]. From the “so that” section, CDS benefit measures were derived. Detailed acceptance criteria were elaborated through ensuing conversations. We estimated user story size with “story points,” and depicted multiple user stories with a use case diagram or feature breakdown structure. Large user stories were split to fit into 2-week iterations.

Results

One example user story was: As a rheumatologist, I want to be advised if my patient with rheumatoid arthritis is not on a disease-modifying anti-rheumatic drug (DMARD), so that they receive optimal therapy and can experience symptom improvement. This yielded a process measure (DMARD use), and an outcome measure (Clinical Disease Activity Index). Following implementation, the DMARD nonuse rate decreased from 3.7% to 1.4%. Patients with a high Clinical Disease Activity Index improved from 13.7% to 7%. For a thromboembolism prevention CDS project, diagrams organized multiple user stories.

Discussion

User stories written in the clinician’s voice aid CDS governance and lead naturally to measures of CDS effectiveness. Estimation of relative story size helps plan CDS delivery dates. User stories prove to be practical even on larger projects.

Conclusions

User stories concisely communicate the who, what, and why of a CDS request, and serve as lightweight requirements for agile development to meet the demand for increasingly diverse CDS.

Giving Your Electronic Health Record a Checkup After COVID-19: A Practical Framework for Reviewing Clinical Decision Support in Light of the Telemedicine Expansion

BACKGROUND

The transformation of health care during COVID-19, with the rapid expansion of telemedicine visits, presents new challenges to chronic care and preventive health providers. Clinical decision support (CDS) is critically important to chronic care providers, and CDS malfunction is common during times of change. It is essential to regularly reassess an organization's ambulatory CDS program to maintain care quality. This is especially true after an immense change, like the COVID-19 telemedicine expansion.

OBJECTIVE

Our objective is to reassess the ambulatory CDS program at a large academic medical center in light of telemedicine's expansion in response to the COVID-19 pandemic.

METHODS

Our clinical informatics team devised a practical framework for an intrapandemic ambulatory CDS assessment focused on the impact of the telemedicine expansion. This assessment began with a quantitative analysis comparing CDS alert performance in the context of in-person and telemedicine visits. Board-certified physician informaticists then completed a formal workflow review of alerts with inferior performance in telemedicine visits. Informaticists then reported on themes and optimization opportunities through the existing CDS governance structure.

RESULTS

Our assessment revealed that 10 of our top 40 alerts by volume were not firing as expected in telemedicine visits. In 3 of the top 5 alerts, providers were significantly less likely to take action in telemedicine when compared to office visits. Cumulatively, alerts in telemedicine encounters had an action taken rate of 5.3% (3257/64,938) compared to 8.3% (19,427/233,636) for office visits. Observations from a clinical informaticist workflow review included the following: (1) Telemedicine visits have different workflows than office visits. Some alerts developed for the office were not appearing at the optimal time in the telemedicine workflow. (2) Missing clinical data is a common reason for the decreased alert firing seen in telemedicine visits. (3) Remote patient monitoring and patient-reported clinical data entered through the portal could replace data collection usually completed in the office by a medical assistant or registered nurse.

CONCLUSIONS

In a large academic medical center at the pandemic epicenter, an intrapandemic ambulatory CDS assessment revealed clinically significant CDS malfunctions that highlight the importance of reassessing ambulatory CDS performance after the telemedicine expansion.

Transforming primary medical research knowledge into clinical decision

While the utility of computerized clinical decision support (CCDS) for multiple select clinical domains has been clearly demonstrated, much less is known about the full breadth of domains to which CCDS approaches could be productively applied. To explore the applicability of CCDS to general medical knowledge, we sampled a total of 500 primary research articles from 4 high-impact medical journals. Employing rule-based templates, we created high-level CCDS rules for 72% (361/500) of primary medical research articles. We subsequently identified data sources needed to implement those rules. Ourfindings suggest that CCDS approaches, perhaps in the form of non-interruptive infobuttons, could be much more broadly applied. In addition, our analytic methods appear to provide a means of prioritizing and quantitating the relative utility of available data sources for purposes of CCDS.

Predictive models for cochlear implant outcomes: Performance, generalizability, and the impact of cohort size

While cochlear implants have helped hundreds of thousands of individuals, it remains difficult to predict the extent to which an individual's hearing will benefit from implantation. Several publications indicate that machine learning may improve predictive accuracy of cochlear implant outcomes compared to classical statistical methods. However, existing studies are limited in terms of model validation and evaluating factors like sample size on predictive performance. We conduct a thorough examination of machine learning approaches to predict word recognition scores (WRS) measured approximately 12 months after implantation in adults with post-lingual hearing loss. This is the largest retrospective study of cochlear implant outcomes to date, evaluating 2,489 cochlear implant recipients from three clinics. We demonstrate that while machine learning models significantly outperform linear models in prediction of WRS, their overall accuracy remains limited (mean absolute error: 17.9-21.8). The models are robust across clinical cohorts, with predictive error increasing by at most 16% when evaluated on a clinic excluded from the training set. We show that predictive improvement is unlikely to be improved by increasing sample size alone, with doubling of sample size estimated to only increasing performance by 3% on the combined dataset. Finally, we demonstrate how the current models could support clinical decision making, highlighting that subsets of individuals can be identified that have a 94% chance of improving WRS by at least 10% points after implantation, which is likely to be clinically meaningful. We discuss several implications of this analysis, focusing on the need to improve and standardize data collection.

A qualitative study of prescribing errors among multi-professional prescribers within an e-prescribing system

Background Computerised Physician Order Entry (CPOE) is considered to enhance the safety of prescribing. However, it can have unintended consequences and new forms of prescribing error have been reported. Objective The aim of this study was to explore the causes and contributing factors associated with prescribing errors reported by multidisciplinary prescribers working within a CPOE system. Main Outcome Measure Multidisciplinary prescribers experience of prescribing errors in an CPOE system. Method This qualitative study was conducted in a hospital with a well-established CPOE system. Semi-structured qualitative interviews were conducted with prescribers from the professions of pharmacy, nursing, and medicine. Interviews analysed using a mixed inductive and deductive approach to develop a framework for the causes of error. Results Twenty-three prescribers were interviewed. Six main themes influencing prescribing were found: the system, the prescriber, the patient, the team, the task of prescribing and the work environment. Prominent issues related to CPOE included, incorrect drug name picking, default auto-population of dosages, alert fatigue and remote prescribing. These interacted within a complex prescribing environment. No substantial differences in the experience of CPOE were found between the professions. Conclusion Medical and non-medical prescribers have similar experiences of prescribing errors when using CPOE, aligned with existing published literature about medical prescribing. Causes of electronic prescribing errors are multifactorial in nature and prescribers describe how factors interact to create the conditions errors. While interventions should focus on direct CPOE issues, such as training and design, socio-technical, and environmental aspects of practice remain important.

Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review

BACKGROUND

The clinical decision support system (CDSS) has become an indispensable tool for reducing medication errors and adverse drug events. However, numerous studies have reported that CDSS alerts are often overridden. The increase in override rates has raised questions about the appropriateness of CDSS application along with concerns about patient safety and quality of care.

OBJECTIVE

The aim of this study was to conduct a systematic review to examine the override rate, the reasons for the alert override at the time of prescribing, and evaluate the appropriateness of overrides.

METHODS

We searched electronic databases, including Google Scholar, PubMed, Embase, Scopus, and Web of Science, without language restrictions between January 1, 2000 and March 31, 2019. Two authors independently extracted data and crosschecked the extraction to avoid errors. The quality of the included studies was examined following Cochrane guidelines.

RESULTS

We included 23 articles in our systematic review. The range of average override alerts was 46.2%-96.2%. An average of 29.4%-100% of the overrides alerts were classified as appropriate, and the rate of appropriateness varied according to the alert type (drug-allergy interaction 63.4%-100%, drug-drug interaction 0%-95%, dose 43.9%-88.8%, geriatric 14.3%-57%, renal 27%-87.5%). The interrater reliability for the assessment of override alerts appropriateness was excellent (kappa=0.79-0.97). The most common reasons given for the override were "will monitor" and "patients have tolerated before."

CONCLUSIONS

The findings of our study show that alert override rates are high, and certain categories of overrides such as drug-drug interaction, renal, and geriatric were classified as inappropriate. Nevertheless, large proportions of drug duplication, drug-allergy, and formulary alerts were appropriate, suggesting that these groups of alerts can be primary targets to revise and update the system for reducing alert fatigue. Future efforts should also focus on optimizing alert types, providing clear information, and explaining the rationale of the alert so that essential alerts are not inappropriately overridden.

Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow

The overwhelming number of medication alerts generated by clinical decision support systems (CDSS) has led to inappropriate alert overrides, which may lead to unintended patient harm. This review highlights the factors affecting the alert appropriateness of CDSS and barriers to the fit of CDSS alert with clinical workflow. A literature review was conducted to identify features and functions pertinent to CDSS alert appropriateness using the five rights of CDSS. Moreover, a process improvement method, namely, Lean, was used as a tool to optimise clinical workflows, and the appropriate design for CDSS alert using a human automation interaction (HAI) model was recommended. Evaluating the appropriateness of CDSS alert and its impact on workflow provided insights into how alerts can be designed and triggered effectively to support clinical workflow. The application of Lean methods and tools to analyse alert efficiencies in supporting workflow in this study provides an in-depth understanding of alert-workflow fit problems and their root cause, which is required for improving CDSS design. The application of the HAI model is recommended in the design of CDSS alerts to support various levels and stages of alert automations, namely, information acquisition and analysis, decision action and action implementation.

Reducing Interruptive Alert Burden Using Quality Improvement Methodology

BACKGROUND

Increased adoption of electronic health records (EHR) with integrated clinical decision support (CDS) systems has reduced some sources of error but has led to unintended consequences including alert fatigue. The "pop-up" or interruptive alert is often employed as it requires providers to acknowledge receipt of an alert by taking an action despite the potential negative effects of workflow interruption. We noted a persistent upward trend of interruptive alerts at our institution and increasing requests for new interruptive alerts.

OBJECTIVES

Using Institute for Healthcare Improvement (IHI) quality improvement (QI) methodology, the primary objective was to reduce the total volume of interruptive alerts received by providers.

METHODS

We created an interactive dashboard for baseline alert data and to monitor frequency and outcomes of alerts as well as to prioritize interventions. A key driver diagram was developed with a specific aim to decrease the number of interruptive alerts from a baseline of 7,250 to 4,700 per week (35%) over 6 months. Interventions focused on the following key drivers: appropriate alert display within workflow, clear alert content, alert governance and standardization, user feedback regarding overrides, and respect for user knowledge.

RESULTS

A total of 25 unique alerts accounted for 90% of the total interruptive alert volume. By focusing on these 25 alerts, we reduced interruptive alerts from 7,250 to 4,400 per week.

CONCLUSION

Systematic and structured improvements to interruptive alerts can lead to overall reduced interruptive alert burden. Using QI methods to prioritize our interventions allowed us to maximize our impact. Further evaluation should be done on the effects of reduced interruptive alerts on patient care outcomes, usability heuristics on cognitive burden, and direct feedback mechanisms on alert utility.

Do computerized clinical decision support systems improve the prescribing of oral anticoagulants? A systematic review

BACKGROUND

Serious adverse drug reactions have been associated with the underuse or the misuse of oral anticoagulant therapy. We systematically reviewed the impact of computerized clinical decision support systems (CDSS) on the prescribing of oral anticoagulants and we described CDSS features associated with success or failure.

METHODS

We searched Medline, Embase, CENTRAL, CINHAL, and PsycINFO for studies that compared CDSS for the initiation or monitoring of oral anticoagulants with routine care. Two reviewers performed study selection, data collection, and risk-of-bias assessment. Disagreements were resolved with a third reviewer. Potentially important CDSS features, identified from previous literature, were evaluated.

RESULTS

Sixteen studies were included in our qualitative synthesis. Most trials were performed in primary care (n = 7) or hospitals (n = 6) and included atrial fibrillation (AF) patients (n = 9). Recommendations mainly focused on anticoagulation underuse (n = 11) and warfarin-drug interactions (n = 5). Most CDSS were integrated in electronic records or prescribing and provided support automatically at the time and location of decision-making. Significant improvements in practitioner performance were found in 9 out of 16 studies, while clinical outcomes were poorly reported. CDSS features seemed slightly more common in studies that demonstrated improvement.

CONCLUSIONS

CDSS might positively impact the use of oral anticoagulants in AF patients at high risk of stroke. The scope of CDSS should now evolve to assist prescribers in selecting the most appropriate and tailored medication. Efforts should nevertheless be made to improve the relevance of notifications and to address implementation outcomes.

Real-Time Automated Hazard Detection Framework for Health Information Technology Systems

An increase in the reliability of Health Information Technology (HIT) will facilitate institutional trust and credibility of the systems. In this paper, we present an end-to-end framework for improving the reliability and performance of HIT systems. Specifically, we describe the system model, present some of the methods that drive the model, and discuss an initial implementation of two of the proposed methods using data from the Veterans Affairs HIT and Corporate Data Warehouse systems. The contributions of this paper, thus, include (1) the design of a system model for monitoring and detecting hazards in HIT systems, (2) a data-driven approach for analysing the health care data warehouse, (3) analytical methods for characterising and analysing failures in HIT systems, and (4) a tool architecture for generating and reporting hazards in HIT systems. Our goal is to work towards an automated system that will help identify opportunities for improvements in HIT systems.

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons

Background

Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.

Objective

Investigate whether user override comments can be used to discover malfunctions.

Methods

We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: “broken,” “not broken, but could be improved,” and “not broken.” We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth.

Results

Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738.

Discussion

Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts.

Conclusion

Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.

Using statistical anomaly detection models to find clinical decision support malfunctions

Objective

Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems.

Methods

We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic® (Epic Systems Corporation, Madison, WI, USA) at Brigham and Women's Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone.

Results

Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone.

Conclusions

Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.

Unintended adverse consequences of a clinical decision support system: two cases

Many institutions have implemented clinical decision support systems (CDSSs). While CDSS research papers have focused on benefits of these systems, there is a smaller body of literature showing that CDSSs may also produce unintended adverse consequences (UACs). Detailed here are 2 cases of UACs resulting from a CDSS. Both of these cases were related to external systems that fed data into the CDSS. In the first case, lack of knowledge of data categorization in an external pharmacy system produced a UAC; in the second case, the change of a clinical laboratory instrument produced the UAC. CDSSs rely on data from many external systems. These systems are dynamic and may have changes in hardware, software, vendors, or processes. Such changes can affect the accuracy of CDSSs. These cases point to the need for the CDSS team to be familiar with these external systems. This team (manager and alert builders) should include members in specific clinical specialties with deep knowledge of these external systems.

Importance of clinical decision support system response time monitoring: a case report

Clinical decision support (CDS) systems are prevalent in electronic health records and drive many safety advantages. However, CDS systems can also cause unintended consequences. Monitoring programs focused on alert firing rates are important to detect anomalies and ensure systems are working as intended. Monitoring efforts do not generally include system load and time to generate decision support, which is becoming increasingly important as more CDS systems rely on external, web-based content and algorithms. We report a case in which a web-based service caused significant increase in the time to generate decision support, in turn leading to marked delays in electronic health record system responsiveness, which could have led to patient safety events. Given this, it is critical to consider adding decision support-time generation to ongoing CDS system monitoring programs.

Towards a Maturity Model for Clinical Decision Support Operations

Clinical decision support (CDS) systems delivered through the electronic health record are an important element of quality and safety initiatives within a health care system. However, managing a large CDS knowledge base can be an overwhelming task for informatics teams. Additionally, it can be difficult for these informatics teams to communicate their goals with external operational stakeholders and define concrete steps for improvement. We aimed to develop a maturity model that describes a roadmap toward organizational functions and processes that help health care systems use CDS more effectively to drive better outcomes. We developed a maturity model for CDS operations through discussions with health care leaders at 80 organizations, iterative model development by four clinical informaticists, and subsequent review with 19 health care organizations. We ceased iterations when feedback from three organizations did not result in any changes to the model. The proposed CDS maturity model includes three main "pillars": "Content Creation," "Analytics and Reporting," and "Governance and Management." Each pillar contains five levels-advancing along each pillar provides CDS teams a deeper understanding of the processes CDS systems are intended to improve. A "roof" represents the CDS functions that become attainable after advancing along each of the pillars. Organizations are not required to advance in order and can develop in one pillar separately from another. However, we hypothesize that optimal deployment of preceding levels and advancing in tandem along the pillars increase the value of organizational investment in higher levels of CDS maturity. In addition to describing the maturity model and its development, we also provide three case studies of health care organizations using the model for self-assessment and determine next steps in CDS development.

A practical computerized decision support system for predicting the severity of Alzheimer's disease of an individual

Computerized clinical decision support systems can help to provide objective, standardized, and timely dementia diagnosis. However, current computerized systems are mainly based on group analysis, discrete classification of disease stages, or expensive and not readily accessible biomarkers, while current clinical practice relies relatively heavily on cognitive and functional assessments (CFA). In this study, we developed a computational framework using a suite of machine learning tools for identifying key markers in predicting the severity of Alzheimer's disease (AD) from a large set of biological and clinical measures. Six machine learning approaches, namely Kernel Ridge Regression (KRR), Support Vector Regression, and k-Nearest Neighbor for regression and Support Vector Machine (SVM), Random Forest, and k-Nearest Neighbor for classification, were used for the development of predictive models. We demonstrated high predictive power of CFA. Predictive performance of models incorporating CFA was shown to consistently have higher accuracy than those based solely on biomarker modalities. We found that KRR and SVM were the best performing regression and classification methods respectively. The optimal SVM performance was observed for a set of four CFA test scores (FAQ, ADAS13, MoCA, MMSE) with multi-class classification accuracy of 83.0%, 95%CI = (72.1%, 93.8%) while the best performance of the KRR model was reported with combined CFA and MRI neuroimaging data, i.e., R ² = 0.874, 95%CI = (0.827, 0.922). Given the high predictive power of CFA and their widespread use in clinical practice, we then designed a data-driven and self-adaptive computerized clinical decision support system (CDSS) prototype for evaluating the severity of AD of an individual on a continuous spectrum. The system implemented an automated computational approach for data pre-processing, modelling, and validation and used exclusively the scores of selected cognitive measures as data entries. Taken together, we have developed an objective and practical CDSS to aid AD diagnosis.

Creating an immunization content database for knowledge management across clinical systems

Purpose

A initiative at an academic medical center to create a single database of immunization-related content to inform the build and configuration of immunization-related knowledge assets across multiple clinical systems is described.

Methods

Semistructured expert interviews were conducted to ascertain the immunization information needs of the institution’s clinical systems. Based on those needs, an immunization domain model constructed with data available from the Centers for Disease Control and Prevention (CDC) website was developed and used to analyze and compare current immunization-related content from CDC data sources with the content of the institution’s clinical systems.

Results

Five identified clinical systems that used immunization-related content collectively required 22 unique information concepts, 11 of which were obtainable from CDC vaccine code sets. The proportion of vaccines designated by CDC as active products (i.e., currently available administrable vaccines) that were included in the 5 clinical systems ranged from 59% to 95%; in addition, some non–active-status vaccines were listed as active-status products in the various clinical systems. Upon further review, updates to immunization-related content in the 5 clinical systems were implemented.

Conclusion

Creating a single database for immunization-related content based on CDC data facilitated an explicit and tractable knowledge management process and helped ensure that clinical systems had correct and current content. The immunization domain model created has the potential to assist in the automated detection of updates and relaying those updates to the applicable clinical systems.

Impact of Clinicians' Use of Electronic Knowledge Resources on Clinical and Learning Outcomes: Systematic Review and Meta-Analysis

BACKGROUND

Clinicians use electronic knowledge resources, such as Micromedex, UpToDate, and Wikipedia, to deliver evidence-based care and engage in point-of-care learning. Despite this use in clinical practice, their impact on patient care and learning outcomes is incompletely understood. A comprehensive synthesis of available evidence regarding the effectiveness of electronic knowledge resources would guide clinicians, health care system administrators, medical educators, and informaticians in making evidence-based decisions about their purchase, implementation, and use.

OBJECTIVE

The aim of this review is to quantify the impact of electronic knowledge resources on clinical and learning outcomes.

METHODS

We searched MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published from 1991 to 2017. Two authors independently screened studies for inclusion and extracted outcomes related to knowledge, skills, attitudes, behaviors, patient effects, and cost. We used random-effects meta-analysis to pool standardized mean differences (SMDs) across studies.

RESULTS

Of 10,811 studies screened, we identified 25 eligible studies published between 2003 and 2016. A total of 5 studies were randomized trials, 22 involved physicians in practice or training, and 10 reported potential conflicts of interest. A total of 15 studies compared electronic knowledge resources with no intervention. Of these, 7 reported clinician behaviors, with a pooled SMD of 0.47 (95% CI 0.27 to 0.67; P<.001), and 8 reported objective patient effects with a pooled SMD of 0.19 (95% CI 0.07 to 0.32; P=.003). Heterogeneity was large (I2>50%) across studies. When compared with other resources-7 studies, not amenable to meta-analytic pooling-the use of electronic knowledge resources was associated with increased frequency of answering questions and perceived benefits on patient care, with variable impact on time to find an answer. A total of 2 studies compared different implementations of the same electronic knowledge resource.

CONCLUSIONS

Use of electronic knowledge resources is associated with a positive impact on clinician behaviors and patient effects. We found statistically significant associations between the use of electronic knowledge resources and improved clinician behaviors and patient effects. When compared with other resources, the use of electronic knowledge resources was associated with increased success in answering clinical questions, with variable impact on speed. Comparisons of different implementation strategies of the same electronic knowledge resource suggest that there are benefits from allowing clinicians to choose to access the resource, versus automated display of resource information, and from integrating patient-specific information. A total of 4 studies compared different commercial electronic knowledge resources, with variable results. Resource implementation strategies can significantly influence outcomes but few studies have examined such factors.

[Artificial intelligence in medicine-the wrong track or promise of cure?]

Artificial intelligence (AI) has attained a new level of maturity in recent years and is developing into the driver of digitalization in all areas of life. AI is a cross-sectional technology with great importance for all branches of medicine employing imaging as well as text and biodata. There is no field of medicine that remains unaffected by AI, with AI-assisted clinical decision-making assuming a particularly important role. AI methods are becoming established in medial workflow management and for prediction of therapeutic success or treatment outcome. AI systems are already able to lend support to imaging-based diagnosis and patient management, but cannot suggest critical decisions. The corresponding preventive or therapeutic measures can be more rationally assessed with the help of AI, although the number of diseases covered is currently far too low for the creation of robust systems for clinical routine. Prerequisite for the comprehensive use of AI systems is appropriate training to enable physicians to decide when computer-assisted decision-making can be relied upon.

A Prescription For Enhancing Electronic Prescribing Safety

While electronic prescribing has been shown to reduce medication errors and improve prescribing safety, it is vulnerable to error-prone processes. We review six intersecting areas in which changes to electronic prescribing systems, particularly in the outpatient setting, could transform medication ordering quality and safety. We recommend incorporating medication indications into electronic prescribing, establishing a single shared online medication list, implementing the transmission of electronic cancellation orders to pharmacies (CancelRx) to ensure that drugs are safely and reliably discontinued, implementing standardized structured and codified prescription instructions, reengineering clinical decision support, and redesigning electronic prescribing to facilitate the ordering of nondrug alternatives.

A Systematic Review of Wearable Sensors and IoT-Based Monitoring Applications for Older Adults - a Focus on Ageing Population and Independent Living

This review aims to present current advancements in wearable technologies and IoT-based applications to support independent living. The secondary aim was to investigate the barriers and challenges of wearable sensors and Internet-of-Things (IoT) monitoring solutions for older adults. For this work, we considered falls and activity of daily life (ADLs) for the ageing population (older adults). A total of 327 articles were screened, and 14 articles were selected for this review. This review considered recent studies published between 2015 and 2019. The research articles were selected based on the inclusion and exclusion criteria, and studies that support or present a vision to provide advancement to the current space of ADLs, independent living and supporting the ageing population. Most studies focused on the system aspects of wearable sensors and IoT monitoring solutions including advanced sensors, wireless data collection, communication platform and usability. Moderate to low usability/ user-friendly approach is reported in most of the studies. Other issues found were inaccurate sensors, battery/ power issues, restricting the users within the monitoring area/ space and lack of interoperability. The advancement of wearable technology and the possibilities of using advanced IoT technology to assist older adults with their ADLs and independent living is the subject of many recent research and investigation.

Systems engineering and human factors support of a system of novel EHR-integrated tools to prevent harm in the hospital

We established a Patient Safety Learning Laboratory comprising 2 core and 3 individual project teams to introduce a suite of digital health tools integrated with our electronic health record to identify, assess, and mitigate threats to patient safety in real time. One of the core teams employed systems engineering (SE) and human factors (HF) methods to analyze problems, design and develop improvements to intervention components, support implementation, and evaluate the system of systems as an integrated whole. Of the 29 participants, 19 and 16 participated in surveys and focus groups, respectively, about their perception of SE and HF. We identified 7 themes regarding use of the 12 SE and HF methods over the 4-year project. Qualitative methods (interviews, focus, groups, observations, usability testing) were most frequently used, typically by individual project teams, and generated the most insight. Quantitative methods (failure mode and effects analysis, simulation modeling) typically were used by the SE and HF core team but generated variable insight. A decentralized project structure led to challenges using these SE and HF methods at the project and systems level. We offer recommendations and insights for using SE and HF to support digital health patient safety initiatives.

Assessing the Safety of Custom Web-Based Clinical Decision Support Systems in Electronic Health Records: A Case Study

BACKGROUND

With the widespread adoption of vendor-supplied electronic health record (EHR) systems, clinical decision support (CDS) customization efforts beyond those anticipated by the vendor may require the use of technologies external to the EHR such as web services. Pursuing such customizations, however, is not without risk. Validating the expected behavior of a customized CDS system in the high-volume, complex environment of the live EHR is a challenging problem.

OBJECTIVE

This article identifies technology failures that impacted clinical care related to web service-based advanced custom CDS systems embedded in the complex sociotechnical context of a production EHR.

METHODS

In an academic health system's primary care network, we performed an inventory of incidents between January 1, 2008 and December 31, 2016 related to a customized CDS system and performed a targeted review of changes in the CDS source code. Additional feedback on the root cause of individual incidents was obtained through interviews with members of the CDS project teams.

RESULTS

We identified five CDS malfunctions that impaired clinical workflow. The mechanisms for these failures are mapped to four characteristics of well-behaved applications: (1) system integrity; (2) data integrity; (3) reliability; and (4) scalability. Over the 9-year period, two malfunctions of the customized CDS significantly impaired clinical workflow for a total of 5 hours. Lesser impacts-loss of individual features with straightforward workarounds-arose from three malfunctions, which affected users on 53 days.

DISCUSSION

Advanced customization of EHRs for the purpose of CDS can present significant risks to clinical workflow.

CONCLUSION

This case study highlights that advanced customization of CDS within a commercial EHR may support care for complex patient populations, but ongoing monitoring and support is required to ensure its safe use.

Reduced Verification of Medication Alerts Increases Prescribing Errors

OBJECTIVE

Clinicians using clinical decision support (CDS) to prescribe medications have an obligation to ensure that prescriptions are safe. One option is to verify the safety of prescriptions if there is uncertainty, for example, by using drug references. Supervisory control experiments in aviation and process control have associated errors, with reduced verification arising from overreliance on decision support. However, it is unknown whether this relationship extends to clinical decision-making. Therefore, we examine whether there is a relationship between verification behaviors and prescribing errors, with and without CDS medication alerts, and whether task complexity mediates this.

METHODS

A total of 120 students in the final 2 years of a medical degree prescribed medicines for patient scenarios using a simulated electronic prescribing system. CDS (correct, incorrect, and no CDS) and task complexity (low and high) were varied. Outcomes were omission (missed prescribing errors) and commission errors (accepted false-positive alerts). Verification measures were access of drug references and view time percentage of task time.

RESULTS

Failure to access references for medicines with prescribing errors increased omission errors with no CDS (high-complexity: χ ²(1) = 12.716; p < 0.001) and incorrect CDS (Fisher's exact; low-complexity: p = 0.002; high-complexity: p = 0.001). Failure to access references for false-positive alerts increased commission errors (low-complexity: χ ²(1) = 16.673, p < 0.001; high-complexity: χ ²(1) = 18.690, p < 0.001). Fewer participants accessed relevant references with incorrect CDS compared with no CDS (McNemar; low-complexity: p < 0.001; high-complexity: p < 0.001). Lower view time percentages increased omission (F(3, 361.914) = 4.498; p = 0.035) and commission errors (F(1, 346.223) = 2.712; p = 0.045). View time percentages were lower in CDS-assisted conditions compared with unassisted conditions (F(2, 335.743) = 10.443; p < 0.001).

DISCUSSION

The presence of CDS reduced verification of prescription safety. When CDS was incorrect, reduced verification was associated with increased prescribing errors.

CONCLUSION

CDS can be incorrect, and verification provides one mechanism to detect errors. System designers need to facilitate verification without increasing workload or eliminating the benefits of correct CDS.

Current challenges in health information technology-related patient safety

We identify and describe nine key, short-term, challenges to help healthcare organizations, health information technology developers, researchers, policymakers, and funders focus their efforts on health information technology-related patient safety. Categorized according to the stage of the health information technology lifecycle where they appear, these challenges relate to (1) developing models, methods, and tools to enable risk assessment; (2) developing standard user interface design features and functions; (3) ensuring the safety of software in an interfaced, network-enabled clinical environment; (4) implementing a method for unambiguous patient identification (1-4 Design and Development stage); (5) developing and implementing decision support which improves safety; (6) identifying practices to safely manage information technology system transitions (5 and 6 Implementation and Use stage); (7) developing real-time methods to enable automated surveillance and monitoring of system performance and safety; (8) establishing the cultural and legal framework/safe harbor to allow sharing information about hazards and adverse events; and (9) developing models and methods for consumers/patients to improve health information technology safety (7-9 Monitoring, Evaluation, and Optimization stage). These challenges represent key "to-do's" that must be completed before we can expect to have safe, reliable, and efficient health information technology-based systems required to care for patients.

Reduced Effectiveness of Interruptive Drug-Drug Interaction Alerts after Conversion to a Commercial Electronic Health Record

BACKGROUND

Drug-drug interaction (DDI) alerts in electronic health records (EHRs) can help prevent adverse drug events, but such alerts are frequently overridden, raising concerns about their clinical usefulness and contribution to alert fatigue.

OBJECTIVE

To study the effect of conversion to a commercial EHR on DDI alert and acceptance rates.

DESIGN

Two before-and-after studies.

PARTICIPANTS

3277 clinicians who received a DDI alert in the outpatient setting.

INTERVENTION

Introduction of a new, commercial EHR and subsequent adjustment of DDI alerting criteria.

MAIN MEASURES

Alert burden and proportion of alerts accepted.

KEY RESULTS

Overall interruptive DDI alert burden increased by a factor of 6 from the legacy EHR to the commercial EHR. The acceptance rate for the most severe alerts fell from 100 to 8.4%, and from 29.3 to 7.5% for medium severity alerts (P < 0.001). After disabling the least severe alerts, total DDI alert burden fell by 50.5%, and acceptance of Tier 1 alerts rose from 9.1 to 12.7% (P < 0.01).

CONCLUSIONS

Changing from a highly tailored DDI alerting system to a more general one as part of an EHR conversion decreased acceptance of DDI alerts and increased alert burden on users. The decrease in acceptance rates cannot be fully explained by differences in the clinical knowledge base, nor can it be fully explained by alert fatigue associated with increased alert burden. Instead, workflow factors probably predominate, including timing of alerts in the prescribing process, lack of differentiation of more and less severe alerts, and features of how users interact with alerts.

Smashing the strict hierarchy: three cases of clinical decision support malfunctions involving carvedilol

Clinical vocabularies allow for standard representation of clinical concepts, and can also contain knowledge structures, such as hierarchy, that facilitate the creation of maintainable and accurate clinical decision support (CDS). A key architectural feature of clinical hierarchies is how they handle parent-child relationships - specifically whether hierarchies are strict hierarchies (allowing a single parent per concept) or polyhierarchies (allowing multiple parents per concept). These structures handle subsumption relationships (ie, ancestor and descendant relationships) differently. In this paper, we describe three real-world malfunctions of clinical decision support related to incorrect assumptions about subsumption checking for β-blocker, specifically carvedilol, a non-selective β-blocker that also has α-blocker activity. We recommend that 1) CDS implementers should learn about the limitations of terminologies, hierarchies, and classification, 2) CDS implementers should thoroughly test CDS, with a focus on special or unusual cases, 3) CDS implementers should monitor feedback from users, and 4) electronic health record (EHR) and clinical content developers should offer and support polyhierarchical clinical terminologies, especially for medications.