How intervention studies measure the effectiveness of medication safety-related clinical decision support systems in primary and long-term care: a systematic review

BACKGROUND

Medication errors and associated adverse drug events (ADE) are a major cause of morbidity and mortality worldwide. In recent years, the prevention of medication errors has become a high priority in healthcare systems. In order to improve medication safety, computerized Clinical Decision Support Systems (CDSS) are increasingly being integrated into the medication process. Accordingly, a growing number of studies have investigated the medication safety-related effectiveness of CDSS. However, the outcome measures used are heterogeneous, leading to unclear evidence. The primary aim of this study is to summarize and categorize the outcomes used in interventional studies evaluating the effects of CDSS on medication safety in primary and long-term care.

METHODS

We systematically searched PubMed, Embase, CINAHL, and Cochrane Library for interventional studies evaluating the effects of CDSS targeting medication safety and patient-related outcomes. We extracted methodological characteristics, outcomes and empirical findings from the included studies. Outcomes were assigned to three main categories: process-related, harm-related, and cost-related. Risk of bias was assessed using the Evidence Project risk of bias tool.

RESULTS

Thirty-two studies met the inclusion criteria. Almost all studies (n = 31) used process-related outcomes, followed by harm-related outcomes (n = 11). Only three studies used cost-related outcomes. Most studies used outcomes from only one category and no study used outcomes from all three categories. The definition and operationalization of outcomes varied widely between the included studies, even within outcome categories. Overall, evidence on CDSS effectiveness was mixed. A significant intervention effect was demonstrated by nine of fifteen studies with process-related primary outcomes (60%) but only one out of five studies with harm-related primary outcomes (20%). The included studies faced a number of methodological problems that limit the comparability and generalizability of their results.

CONCLUSIONS

Evidence on the effectiveness of CDSS is currently inconclusive due in part to inconsistent outcome definitions and methodological problems in the literature. Additional high-quality studies are therefore needed to provide a comprehensive account of CDSS effectiveness. These studies should follow established methodological guidelines and recommendations and use a comprehensive set of harm-, process- and cost-related outcomes with agreed-upon and consistent definitions.

PROSPERO REGISTRATION

CRD42023464746.

Protocol for an intervention development and pilot implementation evaluation study of an e-health solution to improve newborn care quality and survival in two low-resource settings, Malawi and Zimbabwe: Neotree

INTRODUCTION

Every year 2.4 million deaths occur worldwide in babies younger than 28 days. Approximately 70% of these deaths occur in low-resource settings because of failure to implement evidence-based interventions. Digital health technologies may offer an implementation solution. Since 2014, we have worked in Bangladesh, Malawi, Zimbabwe and the UK to develop and pilot Neotree: an android app with accompanying data visualisation, linkage and export. Its low-cost hardware and state-of-the-art software are used to improve bedside postnatal care and to provide insights into population health trends, to impact wider policy and practice.

METHODS AND ANALYSIS

This is a mixed methods (1) intervention codevelopment and optimisation and (2) pilot implementation evaluation (including economic evaluation) study. Neotree will be implemented in two hospitals in Zimbabwe, and one in Malawi. Over the 2-year study period clinical and demographic newborn data will be collected via Neotree, in addition to behavioural science informed qualitative and quantitative implementation evaluation and measures of cost, newborn care quality and usability. Neotree clinical decision support algorithms will be optimised according to best available evidence and clinical validation studies.

ETHICS AND DISSEMINATION

This is a Wellcome Trust funded project (215742_Z_19_Z). Research ethics approvals have been obtained: Malawi College of Medicine Research and Ethics Committee (P.01/20/2909; P.02/19/2613); UCL (17123/001, 6681/001, 5019/004); Medical Research Council Zimbabwe (MRCZ/A/2570), BRTI and JREC institutional review boards (AP155/2020; JREC/327/19), Sally Mugabe Hospital Ethics Committee (071119/64; 250418/48). Results will be disseminated via academic publications and public and policy engagement activities. In this study, the care for an estimated 15 000 babies across three sites will be impacted.

TRIAL REGISTRATION NUMBER

NCT0512707; Pre-results.

Overcoming Major Barriers to Build Efficient Decision Support Systems in Pharmacovigilance

Many decision support methods and systems in pharmacovigilance are built without explicitly addressing specific challenges that jeopardize their eventual success. We describe two sets of challenges and appropriate strategies to address them. The first are data-related challenges, which include using extensive multi-source data of poor quality, incomplete information integration, and inefficient data visualization. The second are user-related challenges, which encompass users' overall expectations and their engagement in developing automated solutions. Pharmacovigilance decision support systems will need to rely on advanced methods, such as natural language processing and validated mathematical models, to resolve data-related issues and provide properly contextualized data. However, sophisticated approaches will not provide a complete solution if end-users do not actively participate in their development, which will ensure tools that efficiently complement existing processes without creating unnecessary resistance. Our group has already tackled these issues and applied the proposed strategies in multiple projects.

A transfer learning approach to facilitate ComBat-based harmonization of multicentre radiomic features in new datasets

PURPOSE

To facilitate the demonstration of the prognostic value of radiomics, multicenter radiomics studies are needed. Pooling radiomic features of such data in a statistical analysis is however challenging, as they are sensitive to the variability in scanner models, acquisition protocols and reconstruction settings, which is often unavoidable in a multicentre retrospective analysis. A statistical harmonization strategy called ComBat was utilized in radiomics studies to deal with the "center-effect". The goal of the present work was to integrate a transfer learning (TL) technique within ComBat-and recently developed alternate versions of ComBat with improved flexibility (M-ComBat) and robustness (B-ComBat)-to allow the use of a previously determined harmonization transform to the radiomic feature values of new patients from an already known center.

MATERIAL AND METHODS

The proposed TL approach were incorporated in the four versions of ComBat (standard, B, M, and B-M ComBat). The proposed approach was evaluated using a dataset of 189 locally advanced cervical cancer patients from 3 centers, with magnetic resonance imaging (MRI) and positron emission tomography (PET) images, with the clinical endpoint of predicting local failure. The impact performance of the TL approach was evaluated by comparing the harmonization achieved using only parts of the data to the reference (harmonization achieved using all the available data). It was performed through three different machine learning pipelines.

RESULTS

The proposed TL technique was successful in harmonizing features of new patients from a known center in all versions of ComBat, leading to predictive models reaching similar performance as the ones developed using the features harmonized with all the data available.

CONCLUSION

The proposed TL approach enables applying a previously determined ComBat transform to new, previously unseen data.

Accuracy assessment methods for physiological model selection toward evaluation of closed-loop controlled medical devices

Physiological closed-loop controlled (PCLC) medical devices are complex systems integrating one or more medical devices with a patient’s physiology through closed-loop control algorithms; introducing many failure modes and parameters that impact performance. These control algorithms should be tested through safety and efficacy trials to compare their performance to the standard of care and determine whether there is sufficient evidence of safety for their use in real care setting. With this aim, credible mathematical models have been constructed and used throughout the development and evaluation phases of a PCLC medical device to support the engineering design and improve safety aspects. Uncertainties about the fidelity of these models and ambiguities about the choice of measures for modeling performance need to be addressed before a reliable PCLC evaluation can be achieved. This research develops tools for evaluating the accuracy of physiological models and establishes fundamental measures for predictive capability assessment across different physiological models. As a case study, we built a refined physiological model of blood volume (BV) response by expanding an original model we developed in our prior work. Using experimental data collected from 16 sheep undergoing hemorrhage and fluid resuscitation, first, we compared the calibration performance of the two candidate physiological models, i.e., original and refined, using root-mean-squared error (RMSE), Akiake information criterion (AIC), and a new multi-dimensional approach utilizing normalized features extracted from the fitting error. Compared to the original model, the refined model demonstrated a significant improvement in calibration performance in terms of RMSE (9%, P = 0.03) and multi-dimensional measure (48%, P = 0.02), while a comparable AIC between the two models verified that the enhanced calibration performance in the refined model is not due to data over-fitting. Second, we compared the physiological predictive capability of the two models under three different scenarios: prediction of subject-specific steady-state BV response, subject-specific transient BV response to hemorrhage perturbation, and leave-one-out inter-subject BV response. Results indicated enhanced accuracy and predictive capability for the refined physiological model with significantly larger proportion of measurements that were within the prediction envelope in the transient and leave-one-out prediction scenarios (P < 0.02). All together, this study helps to identify and merge new methods for credibility assessment and physiological model selection, leading to a more efficient process for PCLC medical device evaluation.

Development of safety and usability guideline for clinical information system

Clinical information systems (CISs) that do not consider usability and safety could lead to harmful events. Therefore, we aimed to develop a safety and usability guideline of CISs that is comprehensive for both users and developers. And the guideline was categorized to apply actual clinical workflow and work environment.The guideline components were extracted through a systematic review of the articles published between 2000 and 2015, and existing CIS safety and/or usability design guidelines. The guideline components were categorized according to clinical workflow and types of user interface (UI). The contents of the guideline were evaluated and validated by experts with 3 specialties: medical informatics, patient safety, and human engineering.Total 1276 guideline components were extracted through article and guideline review. Of these, 464 guideline components were categorized according to 5 divisions of the clinical workflow: "Data identification and selection," "Document entry," "Order entry," "Clinical decision support and alert," and "Management". While 521 guideline components were categorized according to 4 divisions of UI: UIs related to information process steps, "Perception," "Recognition," "Control," and "Feedback". We developed a guideline draft with 219 detailed guidance for clinical task and 70 for UI. Overall appropriateness and comprehensiveness were proven to achieve more than 90% in experts' survey. However, there were significant differences among the groups of specialties in the judgment of appropriateness (P < .001) and comprehensiveness (P = .038).We developed and verified a safety and usability guideline for CIS that qualifies the requirements of both clinical workflows and usability issues. The developed guideline can be a practical tool to enhance the usability and safety of CISs. Further validation is required by applying the guideline for designing the actual CIS.

Effects of patients' hospital discharge preferences on uptake of clinical decision support

The Centers for Medicare and Medicaid Services identified unplanned hospital readmissions as a critical healthcare quality and cost problem. Improvements in hospital discharge decision-making and post-discharge care are needed to address the problem. Utilization of clinical decision support (CDS) can improve discharge decision-making but little is known about the empirical significance of two opposing problems that can occur: (1) negligible uptake of CDS by providers or (2) over-reliance on CDS and underuse of other information. This paper reports an experiment where, in addition to electronic medical records (EMR), clinical decision-makers are provided subjective reports by standardized patients, or CDS information, or both. Subjective information, reports of being eager or reluctant for discharge, was obtained during examinations of standardized patients, who are regularly employed in medical education, and in our experiment had been given scripts for the experimental treatments. The CDS tool presents discharge recommendations obtained from econometric analysis of data from de-identified EMR of hospital patients. 38 clinical decision-makers in the experiment, who were third and fourth year medical students, discharged eight simulated patient encounters with an average length of stay 8.1 in the CDS supported group and 8.8 days in the control group. When the recommendation was “Discharge,” CDS uptake of “Discharge” recommendation was 20% higher for eager than reluctant patients. Compared to discharge decisions in the absence of patient reports: (i) odds of discharging reluctant standardized patients were 67% lower in the CDS-assisted group and 40% lower in the control (no-CDS) group; whereas (ii) odds of discharging eager standardized patients were 75% higher in the control group and similar in CDS-assisted group. These findings indicate that participants were neither ignoring nor over-relying on CDS.

A Multivariable Prediction Model to Select Colorectal Surgical Patients for Co-Management

INTRODUCTION

Increased life expectancy leads to older and frailer surgical patients. Co-management between medical and surgical specialities has proven favourable in complex situations. Selection of patients for co-management is full of difficulties. The aim of this study was to develop a clinical decision support tool to select surgical patients for co-management.

MATERIAL AND METHODS

Clinical data was collected from patient electronic health records with an ICD-9 code for colorectal surgery from January 2012 to December 2015 at a hospital in Lisbon. The outcome variable consists in co-management signalling. A dataset from 344 patients was used to develop the prediction model and a second data set from 168 patients was used for external validation.

RESULTS

Using logistic regression modelling the authors built a five variable (age, burden of comorbidities, ASA-PS status, surgical risk and recovery time) predictive referral model for co-management. This model has an area under the curve (AUC) of 0.86 (95% CI: 0.81 - 0.90), a predictive Brier score of 0.11, a sensitivity of 0.80, a specificity of 0.82 and an accuracy of 81.3%.

DISCUSSION

Early referral of high-risk patients may be valuable to guide the decision on the best level of post-operative clinical care. We developed a simple bedside decision tool with a good discriminatory and predictive performance in order to select patients for comanagement.

CONCLUSION

A simple bed-side clinical decision support tool of patients for co-management is viable, leading to potential improvement in early recognition and management of postoperative complications and reducing the 'failure to rescue'. Generalizability to other clinical settings requires adequate customization and validation.

[Clinical history in times of big data; a plea for a standard for the structured recording of the clinical history]

Clinical decision support systems to aid the clinician in making a correct diagnosis will only succeed if data from the clinical history are taken into account. However, currently, very little is known on diagnostic test characteristics of specific symptoms, let alone of a pattern of several symptoms with all their cardinal features. We plead for the nation-wide introduction of a standard for the structured recording of the clinical history. To allow for such structured recording, user interfaces of electronic healthcare records must become far more user-friendly. Furthermore, scribes may be used, or, ideally, a digital scribe, a computer application that records the conversation between healthcare professional and patient and creates an automated summary. So far, to our knowledge, no digital scribe encompassing the entire patient history has been implemented into medical practice. We are currently trying to develop such a digital scribe.

Decision support systems for the prediction of lymph node involvement in early breast cancer

The prediction of lymph node involvement represents an important task which could reduce unnecessary surgery and improve the definition of oncological therapies. An artificial intelligence model able to predict it in pre-operative phase requires the interface among multiple data structures. The trade-off among time consuming, expensive and invasive methodologies is emerging in experimental setups exploited for the analysis of sentinel lymph nodes, where machine learning algorithms represent a key ingredient in recorded data elaboration. The accuracy required for clinical applications is obtainable matching different kind of data. Statistical associations of prognostic factors with symptoms and predictive models implemented also through on-line softwares represent useful diagnostic support tools which translate into patients quality of life improvement and costs reduction.

Unexpected Inequality: Disparate-Impact From Artificial Intelligence in Healthcare Decisions

Systemic discrimination in healthcare plagues marginalized groups. Physicians incorrectly view people of color as having high pain tolerance, leading to undertreatment. Women with disabilities are often undiagnosed because their symptoms are dismissed. Low-income patients have less access to appropriate treatment. These patterns, and others, reflect long-standing disparities that have become engrained in U.S. health systems. As the healthcare industry adopts artificial intelligence and algorithminformed (AI) tools, it is vital that regulators address healthcare discrimination. AI tools are increasingly used to make both clinical and administrative decisions by hospitals, physicians, and insurers--yet there is no framework that specifically places nondiscrimination obligations on AI users. The Food and Drug Administration has limited authority to regulate AI and has not sought to incorporate anti-discrimination principles in its guidance. Section 1557 of the Affordable Care Act has not been used to enforce nondiscrimination in healthcare AI and is under-utilized by the Office of Civil Rights. State level protections by medical licensing boards or malpractice liability are similarly untested and have not yet extended nondiscrimination obligations to AI. This Article discusses the role of each legal obligation on healthcare AI and the ways in which each system can improve to address discrimination. It highlights the ways in which industries can self-regulate to set nondiscrimination standards and concludes by recommending standards and creating a super-regulator to address disparate impact by AI. As the world moves towards automation, it is imperative that ongoing concerns about systemic discrimination are removed to prevent further marginalization in healthcare.

Economic Value of Data and Analytics for Health Care Providers: Hermeneutic Systematic Literature Review

BACKGROUND

The benefits of data and analytics for health care systems and single providers is an increasingly investigated field in digital health literature. Electronic health records (EHR), for example, can improve quality of care. Emerging analytics tools based on artificial intelligence show the potential to assist physicians in day-to-day workflows. Yet, single health care providers also need information regarding the economic impact when deciding on potential adoption of these tools.

OBJECTIVE

This paper examines the question of whether data and analytics provide economic advantages or disadvantages for health care providers. The goal is to provide a comprehensive overview including a variety of technologies beyond computer-based patient records. Ultimately, findings are also intended to determine whether economic barriers for adoption by providers could exist.

METHODS

A systematic literature search of the PubMed and Google Scholar online databases was conducted, following the hermeneutic methodology that encourages iterative search and interpretation cycles. After applying inclusion and exclusion criteria to 165 initially identified studies, 50 were included for qualitative synthesis and topic-based clustering.

RESULTS

The review identified 5 major technology categories, namely EHRs (n=30), computerized clinical decision support (n=8), advanced analytics (n=5), business analytics (n=5), and telemedicine (n=2). Overall, 62% (31/50) of the reviewed studies indicated a positive economic impact for providers either via direct cost or revenue effects or via indirect efficiency or productivity improvements. When differentiating between categories, however, an ambiguous picture emerged for EHR, whereas analytics technologies like computerized clinical decision support and advanced analytics predominantly showed economic benefits.

CONCLUSIONS

The research question of whether data and analytics create economic benefits for health care providers cannot be answered uniformly. The results indicate ambiguous effects for EHRs, here representing data, and mainly positive effects for the significantly less studied analytics field. The mixed results regarding EHRs can create an economic barrier for adoption by providers. This barrier can translate into a bottleneck to positive economic effects of analytics technologies relying on EHR data. Ultimately, more research on economic effects of technologies other than EHRs is needed to generate a more reliable evidence base.

The Potential of Clinical Decision Support Systems for Prevention, Diagnosis, and Monitoring of Allergic Diseases

Clinical decision support systems (CDSS) aid health care professionals (HCP) in evaluating large sets of information and taking informed decisions during their clinical routine. CDSS are becoming particularly important in the perspective of precision medicine, when HCP need to consider growing amounts of data to create precise patient profiles for personalized diagnosis, treatment and outcome monitoring. In allergy care, several CDSS are being developed and investigated, mainly for respiratory allergic diseases. Although the proposed solutions address different stakeholders, the majority aims at facilitating evidence-based and shared decision-making, incorporating guidelines, and real-time clinical data. We offer here an overview on existing tools, new developments and novel concepts and discuss the potential of digital CDSS in improving prevention, diagnosis and monitoring of allergic diseases.

Triage of patients with venous and lymphatic diseases during the COVID-19 pandemic - The Venous and Lymphatic Triage and Acuity Scale (VELTAS) : A consensus document of the International Union of Phlebology (UIP), Australasian College of Phlebology (ACP), American Vein and Lymphatic Society (AVLS), American Venous Forum (AVF), European College of Phlebology (ECoP), European Venous Forum (EVF), Interventional Radiology Society of Australasia (IRSA), Latin American Venous Forum, Pan-American Society of Phlebology and Lymphology and the Venous Association of India (VAI) This consensus document has been co-published in Phlebology [DOI: 10.1177/0268355520930884] and Journal of Vascular Surgery: Venous and Lymphatic Disorders [DOI: 10.1016/j.jvsv.2020.05.002]. The publications are identical except for minor stylistic and spelling differences in keeping with each journal's style. The contribution has been published under a Attribution-Non Commercial-No Derivatives 4.0 International (CC BY-NC-ND 4.0), (https://creativecommons.org/licenses/by-nc-nd/4.0/)

The coronavirus disease 2019 (COVID-19) global pandemic has resulted in diversion of healthcare resources to the management of patients infected with SARS-CoV-2 virus. Elective interventions and surgical procedures in most countries have been postponed and operating room resources have been diverted to manage the pandemic. The Venous and Lymphatic Triage and Acuity Scale was developed to provide an international standard to rationalise and harmonise the management of patients with venous and lymphatic disorders or vascular anomalies. Triage urgency was determined based on clinical assessment of urgency with which a patient would require medical treatment or surgical intervention. Clinical conditions were classified into six categories of: (1) venous thromboembolism (VTE), (2) chronic venous disease, (3) vascular anomalies, (4) venous trauma, (5) venous compression and (6) lymphatic disease. Triage urgency was categorised into four groups and individual conditions were allocated to each class of triage. These included (1) medical emergencies (requiring immediate attendance), example massive pulmonary embolism; (2) urgent (to be seen as soon as possible), example deep vein thrombosis; (3) semi-urgent (to be attended to within 30-90 days), example highly symptomatic chronic venous disease, and (4) discretionary/non-urgent- (to be seen within 6-12 months), example chronic lymphoedema. Venous and Lymphatic Triage and Acuity Scale aims to standardise the triage of patients with venous and lymphatic disease or vascular anomalies by providing an international consensus-based classification of clinical categories and triage urgency. The scale may be used during pandemics such as the current COVID-19 crisis but may also be used as a general framework to classify urgency of the listed conditions.

Managing COVID-19 With a Clinical Decision Support Tool in a Community Health Network: Algorithm Development and Validation

BACKGROUND

The coronavirus disease (COVID-19) pandemic has resulted in significant morbidity and mortality; large numbers of patients require intensive care, which is placing strain on health care systems worldwide. There is an urgent need for a COVID-19 disease severity assessment that can assist in patient triage and resource allocation for patients at risk for severe disease.

OBJECTIVE

The goal of this study was to develop, validate, and scale a clinical decision support system and mobile app to assist in COVID-19 severity assessment, management, and care.

METHODS

Model training data from 701 patients with COVID-19 were collected across practices within the Family Health Centers network at New York University Langone Health. A two-tiered model was developed. Tier 1 uses easily available, nonlaboratory data to help determine whether biomarker-based testing and/or hospitalization is necessary. Tier 2 predicts the probability of mortality using biomarker measurements (C-reactive protein, procalcitonin, D-dimer) and age. Both the Tier 1 and Tier 2 models were validated using two external datasets from hospitals in Wuhan, China, comprising 160 and 375 patients, respectively.

RESULTS

All biomarkers were measured at significantly higher levels in patients who died vs those who were not hospitalized or discharged (P<.001). The Tier 1 and Tier 2 internal validations had areas under the curve (AUCs) of 0.79 (95% CI 0.74-0.84) and 0.95 (95% CI 0.92-0.98), respectively. The Tier 1 and Tier 2 external validations had AUCs of 0.79 (95% CI 0.74-0.84) and 0.97 (95% CI 0.95-0.99), respectively.

CONCLUSIONS

Our results demonstrate the validity of the clinical decision support system and mobile app, which are now ready to assist health care providers in making evidence-based decisions when managing COVID-19 patient care. The deployment of these new capabilities has potential for immediate impact in community clinics and sites, where application of these tools could lead to improvements in patient outcomes and cost containment.

Developing a Clinical Decision Support Tool for Appropriate Antibiotic Prescribing in Australian General Practice: A Simulation Study

Background. Inappropriate antibiotic prescribing can lead to antimicrobial resistance and drug side effects. Tools that assist general practitioners (GPs) in prescribing decisions may help to optimize prescribing. The aim of this study was to explore the use, acceptability, and feasibility of a clinical decision support (CDS) tool that incorporates evidence-based guidelines and consumer information that integrates with the electronic medical record (EMR). Methods. Eight GPs completed an interview and brief survey after participating in 2 simulated consultations. The survey consisted of demographic questions, perception of realism and representativeness of consultations, Post-Study System Usability Questionnaire, and System Usability Scale. Qualitative data were analyzed using framework analysis. Video data were reviewed, with length of consultation and time spent using the CDS tool documented. Results. Survey responses indicated that all GPs thought the consultations were "real" and representative of real-life consultations; 7 of 8 GPs were satisfied with usability of the tool. Key qualitative findings included that the tool assisted with clinical decision making and informed appropriate antibiotic prescribing. Accessibility and ease of use, including content (guideline and patient education resources), layout, and format, were key factors that determined whether GPs said that they would access the tool in everyday practice. Integration of the tool at multiple sites within the EMR facilitated access to guidelines and assisted in ensuring that the tool fit the clinical workflow. Conclusion. Our CDS tool was acceptable to GPs. Key features required for the tool were easy navigation, clear and useful guideline content, ability to fit into the clinical workflow, and incorporation into the EMR. Piloting of the tool in general practices to assess the impact and feasibility of use in real-world consultations will now be undertaken.

National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018

IMPORTANCE

Despite the broad adoption of electronic health record (EHR) systems across the continuum of care, safety problems persist.

OBJECTIVE

To measure the safety performance of operational EHRs in hospitals across the country during a 10-year period.

DESIGN, SETTING, AND PARTICIPANTS

This case series included all US adult hospitals nationwide that used the National Quality Forum Health IT Safety Measure EHR computerized physician order entry safety test administered by the Leapfrog Group between 2009 and 2018. Data were analyzed from July 1, 2018 to December 1, 2019.

EXPOSURE

The Health IT Safety Measure test, which uses simulated medication orders that have either injured or killed patients previously to evaluate how well hospital EHRs could identify medication errors with potential for patient harm.

MAIN OUTCOMES AND MEASURES

Descriptive statistics for performance on the assessment test over time were calculated at the overall test score level, type of decision support category level, and EHR vendor level.

RESULTS

Among 8657 hospital-years observed during the study, mean (SD) scores on the overall test increased from 53.9% (18.3%) in 2009 to 65.6% (15.4%) in 2018. Mean (SD) hospital score for the categories representing basic clinical decision support increased from 69.8% (20.8%) in 2009 to 85.6% (14.9%) in 2018. For the categories representing advanced clinical decision support, the mean (SD) score increased from 29.6% (22.4%) in 2009 to 46.1% (21.6%) in 2018. There was considerable variation in test performance by EHR.

CONCLUSIONS AND RELEVANCE

These findings suggest that despite broad adoption and optimization of EHR systems in hospitals, wide variation in the safety performance of operational EHR systems remains across a large sample of hospitals and EHR vendors. Hospitals using some EHR vendors had significantly higher test scores. Overall, substantial safety risk persists in current hospital EHR systems.

Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination

BACKGROUND

Although clinical decision support (CDS) alerts are effective reminders of best practices, their effectiveness is blunted by clinicians who fail to respond to an overabundance of inappropriate alerts. An electronic health record (EHR)-integrated machine learning (ML) algorithm is a potentially powerful tool to increase the signal-to-noise ratio of CDS alerts and positively impact the clinician's interaction with these alerts in general.

OBJECTIVE

This study aimed to describe the development and implementation of an ML-based signal-to-noise optimization system (SmartCDS) to increase the signal of alerts by decreasing the volume of low-value herpes zoster (shingles) vaccination alerts.

METHODS

We built and deployed SmartCDS, which builds personalized user activity profiles to suppress shingles vaccination alerts unlikely to yield a clinician's interaction. We extracted all records of shingles alerts from January 2017 to March 2019 from our EHR system, including 327,737 encounters, 780 providers, and 144,438 patients.

RESULTS

During the 6 weeks of pilot deployment, the SmartCDS system suppressed an average of 43.67% (15,425/35,315) potential shingles alerts (appointments) and maintained stable counts of weekly shingles vaccination orders (326.3 with system active vs 331.3 in the control group; P=.38) and weekly user-alert interactions (1118.3 with system active vs 1166.3 in the control group; P=.20).

CONCLUSIONS

All key statistics remained stable while the system was turned on. Although the results are promising, the characteristics of the system can be subject to future data shifts, which require automated logging and monitoring. We demonstrated that an automated, ML-based method and data architecture to suppress alerts are feasible without detriment to overall order rates. This work is the first alert suppression ML-based model deployed in practice and serves as foundational work in encounter-level customization of alert display to maximize effectiveness.

The impact of a computerised decision support system on antibiotic usage in an English hospital

Background Antimicrobial resistance is correlated with the inappropriate use of antibiotics. Computerised decision support systems may help practitioners to make evidence-based decisions when prescribing antibiotics. Objective This study aimed to evaluate the impact of computerized decision support systems on the volume of antibiotics used. Setting A very large 1200-bed teaching hospital in Birmingham, England. Main outcome measure The primary outcome measure was the defined daily doses/1000 occupied bed-days. Method A retrospective longitudinal study was conducted to examine the impact of computerised decision support systems on the volume of antibiotic use. The study compared two periods: one with computerised decision support systems, which lasted for 2 years versus one without which lasted for 2 years after the withdrawal of computerised decision support systems. Antibiotic use data from June 2012 to June 2016 were analysed (comprising 2 years with computerised decision support systems immediately followed by 2 years where computerised decision support systems had been withdrawn). Regression analysis was applied to assess the change in antibiotic consumption through the period of the study. Result From June 2012 to June 2016, total antibiotic usage increased by 13.1% from 1436 to 1625 defined daily doses/1000 bed-days: this trend of increased antibiotic prescribing was more pronounced following the withdrawal of structured prescribing (computerised decision support systems). There was a difference of means of - 110.14 defined daily doses/1000 bed days of the total usage of antibiotics in the period with and without structured prescribing, and this was statistically significant (p = 0.026). From June 2012 to June 2016, the dominant antibiotic class used was penicillins. The trends for the total consumption of all antibiotics demonstrated an increase of use for all antibiotic classes except for tetracyclines, quinolones, and anti-mycobacterial drugs, whereas aminoglycoside usage remained stable. Conclusion The implementation of computerised decision support systems appears to influence the use of antibiotics by reducing their consumption. Further research is required to determine the specific features of computerised decision support systems, which influence increased higher adoption and uptake of this technology.

Standards-Based Clinical Decision Support Platform to Manage Patients Who Meet Guideline-Based Criteria for Genetic Evaluation of Familial Cancer

PURPOSE

The ubiquitous adoption of electronic health records (EHRs) with family health history (FHH) data provides opportunities for tailoring cancer screening strategies to individuals. We aimed to enable a standards-based clinical decision support (CDS) platform for identifying and managing patients who meet guidelines for genetic evaluation of hereditary cancer.

METHODS

The CDS platform (www.opencds.org) was used to implement algorithms based on the 2018 National Comprehensive Cancer Network guidelines for genetic evaluation of hereditary breast/ovarian and colorectal cancer. The platform was designed to be interfaced with different EHR systems via the Health Level Seven International Fast Healthcare Interoperability Resources standard. The platform was integrated with the Epic EHR and evaluated in a pilot study at an academic health care system.

RESULTS

The CDS platform was executed against a target population of 143,012 patients; 5,245 (3.7%) met criteria for genetic evaluation based on the FHH recorded in the EHR. In a clinical pilot study, genetic counselors attempted to reach out to 71 of the patients. Of those patients, 25 (35%) scheduled an appointment, 10 (14%) declined, 2 (3%) did not need genetic counseling, 7 (10%) said they would consider it in the future, and 27 (38%) were unreachable. To date, 13 (52%) of the scheduled patients completed visits, and 2 (15%) of those were found to have pathogenic variants in cancer predisposition genes.

CONCLUSION

A standards-based CDS platform integrated with EHR systems is a promising population-based approach to identify patients who are appropriate candidates for genetic evaluation of hereditary cancers.