Putting Meaning into Meaningful Use: A Roadmap to Successful Integration of Evidence at the Point of Care

Managing Resource Utilization Cost of Laboratory Tests for Patients on Chemotherapy in Johns Hopkins Aramco Healthcare

Introduction

Laboratory testing is a fundamental diagnostic and prognostic tool to ensure the quality of healthcare, treatment, and responses. This study aimed to evaluate the cost of laboratory tests performed for patients undergoing chemotherapy treatment in the oncology treatment center at Johns Hopkins Aramco Healthcare in Saudi Arabia. Additionally, we aimed to reduce the cost of unnecessary laboratory tests in a 1-year period.

Methods

This was a quality improvement study with a quasi-experimental design using DMAIC methodology. The intervention strategy involved educating staff about adhering to the British Columbia Cancer Agency (BCCA) guidelines when ordering laboratory tests for chemotherapy patients, then integrating those guidelines into the electronic health record system. Data were collected for 200 randomly selected cases with 10 different chemotherapy protocols before and after the intervention. A paired t test was used to analyze differences in mean cost for all laboratory tests and unnecessary testing before and after the intervention.

Results

A significant cost reduction was achieved for unnecessary laboratory tests (77%, p < 0.01) when following the BCCA guidelines. In addition, the mean cost of all laboratory tests (including necessary and unnecessary) was significantly reduced by 45.5% (p = 0.023).

Conclusion

Lean thinking in clinical practice, realized by integrating a standardized laboratory test guided by BCCA guidelines into the electronic health record, significantly reduced financial costs within 1 year, thereby enhancing efficient resource utilization in the organization. This quality improvement project may serve to increase awareness of further efforts to improve resource utilization for other oncology treatment protocols.

Integrating Clinical Decision Support Into Electronic Health Record Systems Using a Novel Platform (EvidencePoint): Developmental Study

BACKGROUND

Through our work, we have demonstrated how clinical decision support (CDS) tools integrated into the electronic health record (EHR) assist providers in adopting evidence-based practices. This requires confronting technical challenges that result from relying on the EHR as the foundation for tool development; for example, the individual CDS tools need to be built independently for each different EHR.

OBJECTIVE

The objective of our research was to build and implement an EHR-agnostic platform for integrating CDS tools, which would remove the technical constraints inherent in relying on the EHR as the foundation and enable a single set of CDS tools that can work with any EHR.

METHODS

We developed EvidencePoint, a novel, cloud-based, EHR-agnostic CDS platform, and we will describe the development of EvidencePoint and the deployment of its initial CDS tools, which include EHR-integrated applications for clinical use cases such as prediction of hospitalization survival for patients with COVID-19, venous thromboembolism prophylaxis, and pulmonary embolism diagnosis.

RESULTS

The results below highlight the adoption of the CDS tools, the International Medical Prevention Registry on Venous Thromboembolism-D-Dimer, the Wells' criteria, and the Northwell COVID-19 Survival (NOCOS), following development, usability testing, and implementation. The International Medical Prevention Registry on Venous Thromboembolism-D-Dimer CDS was used in 5249 patients at the 2 clinical intervention sites. The intervention group tool adoption was 77.8% (4083/5249 possible uses). For the NOCOS tool, which was designed to assist with triaging patients with COVID-19 for hospital admission in the event of constrained hospital resources, the worst-case resourcing scenario never materialized and triaging was never required. As a result, the NOCOS tool was not frequently used, though the EvidencePoint platform's flexibility and customizability enabled the tool to be developed and deployed rapidly under the emergency conditions of the pandemic. Adoption rates for the Wells' criteria tool will be reported in a future publication.

CONCLUSIONS

The EvidencePoint system successfully demonstrated that a flexible, user-friendly platform for hosting CDS tools outside of a specific EHR is feasible. The forthcoming results of our outcomes analyses will demonstrate the adoption rate of EvidencePoint tools as well as the impact of behavioral economics "nudges" on the adoption rate. Due to the EHR-agnostic nature of EvidencePoint, the development process for additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system of EvidencePoint.

Universal EHRs Clinical Decision Support for Thromboprophylaxis in Medical Inpatients: A Cluster Randomized Trial

Background

Thromboprophylaxis for medically ill patients during hospitalization and postdischarge remains underutilized. Clinical decision support (CDS) may address this need if embedded within workflow, interchangeable among electronic health records (EHRs), and anchored on a validated model.

Objectives

The purpose of this study was to assess the clinical impact of a universal EHR-integrated CDS tool based on the International Medical Prevention Registry on Venous Thromboembolism plus D-Dimer venous thromboembolism model.

Methods

This was a cluster randomized trial of 4 tertiary academic hospitals from December 21, 2020 to January 21, 2022. Inpatients over age 60 with key medical illnesses were eligible. We embedded CDS at admission and discharge. Hospitals were randomized to intervention (CDS; n = 2) vs usual care (n = 2) groups. The primary outcome was rate of appropriate thromboprophylaxis. Secondary outcomes included venous, arterial, and total thromboembolism, major bleeding, and all-cause mortality through 30 days postdischarge.

Results

After exclusions, 10,699 of 19,823 patients were analyzed. Intervention group tool adoption was 77.8%. Appropriate thromboprophylaxis was increased at intervention hospitals, both inpatient (80.1% vs 72.5%, OR: 1.52, 95% CI: 1.39-1.67) and at discharge (13.6% vs 7.5%, OR: 1.93, 95% CI: 1.60-2.33). There were fewer venous (2.7% vs 3.3%, OR: 0.80, 95% CI: 0.64-1.00), arterial (0.25% vs 0.70%, OR: 0.35, 95% CI: 0.19-0.67), and total thromboembolisms (2.9% vs 4.0%, OR: 0.71, 95% CI: 0.58-0.88) at intervention hospitals. Major bleeding was rare and did not differ between groups. Mortality was higher at intervention hospitals (9.1% vs 7.0%, OR: 1.32, 95% CI: 1.15-1.53).

Conclusions

EHR-embedded CDS increased appropriate thromboprophylaxis and reduced thromboembolism without increasing major bleeding in medically ill inpatients. Mortality was higher at intervention hospitals.

Medical data integration using HL7 standards for patient's early identification

Integration between information systems is critical, especially in the healthcare domain, since interoperability requirements are related to patients’ data confidentiality, safety, and satisfaction. The goal of this study is to propose a solution based on the integration between queue management solution (QMS) and the electronic medical records (EMR), using Health Level Seven (HL7) protocols and Extensible Markup Language (XML). The proposed solution facilitates the patient’s self-check-in within a healthcare organization in UAE. The solution aims to help in minimizing the waiting times within the outpatient department through early identification of patients who hold the Emirates national ID cards, i.e., whether an Emirati or expatriates. The integration components, solution design, and the custom-designed XML and HL7 messages were clarified in this paper. In addition, the study includes a simulation experiment through control and intervention weeks with 517 valid appointments. The experiment goal was to evaluate the patient’s total journey and each related clinical stage by comparing the “routine-based identification” with the “patient’s self-check-in” processes in case of booked appointments. As a key finding, the proposed solution is efficient and could reduce the “patient’s journey time” by more than 14 minutes and “time to identify” patients by 10 minutes. There was also a significant drop in the waiting time to triage and the time to finish the triage process. In conclusion, the proposed solution is considered innovative and can provide a positive added value for the patient’s whole journey.

Automated model versus treating physician for predicting survival time of patients with metastatic cancer

OBJECTIVE

Being able to predict a patient's life expectancy can help doctors and patients prioritize treatments and supportive care. For predicting life expectancy, physicians have been shown to outperform traditional models that use only a few predictor variables. It is possible that a machine learning model that uses many predictor variables and diverse data sources from the electronic medical record can improve on physicians' performance. For patients with metastatic cancer, we compared accuracy of life expectancy predictions by the treating physician, a machine learning model, and a traditional model.

MATERIALS AND METHODS

A machine learning model was trained using 14 600 metastatic cancer patients' data to predict each patient's distribution of survival time. Data sources included note text, laboratory values, and vital signs. From 2015-2016, 899 patients receiving radiotherapy for metastatic cancer were enrolled in a study in which their radiation oncologist estimated life expectancy. Survival predictions were also made by the machine learning model and a traditional model using only performance status. Performance was assessed with area under the curve for 1-year survival and calibration plots.

RESULTS

The radiotherapy study included 1190 treatment courses in 899 patients. A total of 879 treatment courses in 685 patients were included in this analysis. Median overall survival was 11.7 months. Physicians, machine learning model, and traditional model had area under the curve for 1-year survival of 0.72 (95% CI 0.63-0.81), 0.77 (0.73-0.81), and 0.68 (0.65-0.71), respectively.

CONCLUSIONS

The machine learning model's predictions were more accurate than those of the treating physician or a traditional model.

Nurses' Electronic Medical Record Workarounds in Mental Healthcare Settings

This study aimed to examine nurses' EMR workarounds in mental healthcare settings. Of the 52 nurses invited to participate in this study, 50 nurses (96.1%) completed the survey using the EMR nursing workaround instrument and open-ended questions. The data collected were analyzed using descriptive statistics and Pearson's correlation coefficients. The descriptive data were grouped into four units including the cases, contributing causes, and consequences of EMR workarounds, and suggestions for improving EMR implementation. The results showed scores above an average of 3.0 in all of the EMR workaround items, indicating the considerable involvement of nurses in EMR workarounds. The workarounds related to EMR use were using the physician's login account for medication access, performing retrospective documentation, performing documentation before an expected busy situation, and seeking and entering information on external medications. The workarounds associated with colleagues unfamiliar with EMR use included waiting for, filling in for, teaching, and assisting unskilled colleagues. This study identified the problems, consequences, and suggestions associated with EMR implementation for psychiatric patient care. This study added useful information for the administrative, technical, and clinical improvement of EMR implementation in mental healthcare practice.

Integrating Option Grid Patient Decision Aids in the Epic Electronic Health Record: Case Study at 5 Health Systems

Background

Some researchers argue that the successful implementation of patient decision aids (PDAs) into clinical workflows depends on their integration into electronic health records (EHRs). Anecdotally, we know that EHR integration is a complex and time-consuming task; yet, the process has not been examined in detail. As part of an implementation project, we examined the work involved in integrating an encounter PDA for symptomatic uterine fibroids into Epic EHR systems.

Objective

This study aims to identify the steps and time required to integrate a PDA into the Epic EHR system and examine facilitators and barriers to the integration effort.

Methods

We conducted a case study at 5 academic medical centers in the United States. A clinical champion at each institution liaised with their Epic EHR team to initiate the integration of the uterine fibroid Option Grid PDAs into clinician-facing menus. We scheduled regular meetings with the Epic software analysts and an expert Epic technologist to discuss how best to integrate the tools into Epic for use by clinicians with patients. The meetings were then recorded and transcribed. Two researchers independently coded the transcripts and field notes before categorizing the codes and conducting a thematic analysis to identify the facilitators and barriers to EHR integration. The steps were reviewed and edited by an Epic technologist to ensure their accuracy.

Results

Integrating the uterine fibroid Option Grid PDA into clinician-facing menus required an 18-month timeline and a 6-step process, as follows: task priority negotiation with Epic software teams, security risk assessment, technical review, Epic configuration; troubleshooting, and launch. The key facilitators of the process were the clinical champions who advocated for integration at the institutional level and the presence of an experienced technologist who guided Epic software analysts during the build. Another facilitator was the use of an emerging industry standard app platform (Health Level 7 Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources) as a means of integrating the Option Grid into existing systems. This standard platform enabled clinicians to access the tools by using single sign-on credentials and prevented protected health information from leaving the EHR. Key barriers were the lack of control over the Option Grid product developed by EBSCO (Elton B Stephens Company) Health; the periodic Epic upgrades that can result in a pause on new software configurations; and the unforeseen software problems with Option Grid (ie, inability to print the PDA), which delayed the launch of the PDA.

Conclusions

The integration of PDAs into the Epic EHR system requires a 6-step process and an 18-month timeline. The process required support and prioritization from a clinical champion, guidance from an experienced technologist, and a willing EHR software developer team.

A Clinical Decision Support System (KNOWBED) to Integrate Scientific Knowledge at the Bedside: Development and Evaluation Study

BACKGROUND

The evidence-based medicine (EBM) paradigm requires the development of health care professionals' skills in the efficient search of evidence in the literature, and in the application of formal rules to evaluate this evidence. Incorporating this methodology into the decision-making routine of clinical practice will improve the patients' health care, increase patient safety, and optimize resources use.

OBJECTIVE

The aim of this study is to develop and evaluate a new tool (KNOWBED system) as a clinical decision support system to support scientific knowledge, enabling health care professionals to quickly carry out decision-making processes based on EBM during their routine clinical practice.

METHODS

Two components integrate the KNOWBED system: a web-based knowledge station and a mobile app. A use case (bronchiolitis pathology) was selected to validate the KNOWBED system in the context of the Paediatrics Unit of the Virgen Macarena University Hospital (Seville, Spain). The validation was covered in a 3-month pilot using 2 indicators: usability and efficacy.

RESULTS

The KNOWBED system has been designed, developed, and validated to support clinical decision making in mobility based on standards that have been incorporated into the routine clinical practice of health care professionals. Using this tool, health care professionals can consult existing scientific knowledge at the bedside, and access recommendations of clinical protocols established based on EBM. During the pilot project, 15 health care professionals participated and accessed the system for a total of 59 times.

CONCLUSIONS

The KNOWBED system is a useful and innovative tool for health care professionals. The usability surveys filled in by the system users highlight that it is easy to access the knowledge base. This paper also sets out some improvements to be made in the future.

Economic impact of clinical decision support interventions based on electronic health records

Background

Unnecessary healthcare utilization, non-adherence to current clinical guidelines, or insufficient personalized care are perpetual challenges and remain potential major cost-drivers for healthcare systems around the world. Implementing decision support systems into clinical care is promised to improve quality of care and thereby yield substantial effects on reducing healthcare expenditure. In this article, we evaluate the economic impact of clinical decision support (CDS) interventions based on electronic health records (EHR).

Methods

We searched for studies published after 2014 using MEDLINE, CENTRAL, WEB OF SCIENCE, EBSCO, and TUFTS CEA registry databases that encompass an economic evaluation or consider cost outcome measures of EHR based CDS interventions. Thereupon, we identified best practice application areas and categorized the investigated interventions according to an existing taxonomy of front-end CDS tools.

Results and discussion

Twenty-seven studies are investigated in this review. Of those, twenty-two studies indicate a reduction of healthcare expenditure after implementing an EHR based CDS system, especially towards prevalent application areas, such as unnecessary laboratory testing, duplicate order entry, efficient transfusion practice, or reduction of antibiotic prescriptions. On the contrary, order facilitators and undiscovered malfunctions revealed to be threats and could lead to new cost drivers in healthcare. While high upfront and maintenance costs of CDS systems are a worldwide implementation barrier, most studies do not consider implementation cost. Finally, four included economic evaluation studies report mixed monetary outcome results and thus highlight the importance of further high-quality economic evaluations for these CDS systems.

Conclusion

Current research studies lack consideration of comparative cost-outcome metrics as well as detailed cost components in their analyses. Nonetheless, the positive economic impact of EHR based CDS interventions is highly promising, especially with regard to reducing waste in healthcare.

Covering patient's perspective in case-based critical review articles to improve shared decision making in complex cases

Background

The patient has always been at the centre of the evidence‐based medicine model. Case‐based critical reviews, such as best‐evidence topics, however, are incomplete reflections of the evidence‐based medicine philosophy, because they fail to consider the patient's perspective. We propose a new framework, called the ‘Shared Decision Evidence Summary’ (ShaDES), where the patient's perspective on available treatment options is explicitly included.

Methods

Our framework is grounded in the critical appraisal of a clinical scenario, and the development of a clinical question, including patient characteristics, compared options and outcomes to be improved. Answers to the clinical question are informed by the literature, the evaluation of its quality and its potential usefulness to the clinical scenario. Finally, the evidence synthesis is presented to the patient to facilitate the formulation of an evidence‐informed decision about the treatment options.

Key results

Using three similar but contrasted clinical scenarios of patients with low back pain, we illustrate how considering the patient's preferences on the proposed treatment options impact the bottom line, a synthetic formulation of the answer to the focused question. ShaDES includes clinical and psychosocial components, transformed in a searchable question, with a full search strategy.

Conclusions

ShaDES is a practical framework that may facilitate clinical decisions adapted to psychological, social and other relevant non‐clinical characteristics of patients.

External validation of clinical prediction rules for complications and mortality following Clostridioides difficile infection

Background

Several clinical prediction rules (CPRs) for complications and mortality of Clostridioides difficile infection (CDI) have been developed but only a few have gone through external validation, and none is widely recommended in clinical practice.

Methods

CPRs were identified through a systematic review. We included studies that predicted severe or complicated CDI (cCDI) and mortality, reported at least an internal validation step, and for which data were available with minimal modifications. Data from a multicenter prospective cohort of 1380 adults with confirmed CDI were used for external validation. In this cohort, cCDI occurred in 8% of the patients and 30-day all-cause mortality occurred in 12%. The performance of each tool was assessed using individual outcomes, with the same cut-offs and standard parameters.

Results

Seven CPRs were assessed. Three predictive scores for cCDI showed low sensitivity (25–61%) and positive predictive value (PPV; 9–31%), but moderate specificity (54–90%) and negative predictive value (NPV; 82–95%). One model [using age, white blood cell count (WBC), narcotic use, antacids use, and creatinine ratio > 1.5× the normal level as covariates] showed a probability of 25% of cCDI at the optimal cut-off point with 36% sensitivity and 84% specificity. Two scores for mortality had low sensitivity (4–55%) and PPV (25–31%), and moderate specificity (71–78%) and NPV (87–92%). One predictive model for 30-day all-cause mortality [Charlson comorbidity index, WBC, blood urea nitrogen (BUN), diagnosis in ICU, and delirium] showed an AUC-ROC of 0.74. All other CPRs showed lower AUC values (0.63–0.69). Errors in calibration ranged from 12%- 27%.

Conclusions

Included CPRs showed moderate performance for clinical use in a large validation cohort with a majority of patients infected with ribotype 027 strains and a low rate of cCDI and mortality. These data show that better CPRs need to be developed and validated.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Translating clinical trial results into personalized recommendations by considering multiple outcomes and subjective views

Currently, clinicians rely mostly on population-level treatment effects from RCTs, usually considering the treatment's benefits. This study proposes a process, focused on practical usability, for translating RCT data into personalized treatment recommendations that weighs benefits against harms and integrates subjective perceptions of relative severity. Intensive blood pressure treatment (IBPT) was selected as the test case to demonstrate the suggested process, which was divided into three phases: (1) Prediction models were developed using the Systolic Blood-Pressure Intervention Trial (SPRINT) data for benefits and adverse events of IBPT. The models were externally validated using retrospective Clalit Health Services (CHS) data; (2) Predicted risk reductions and increases from these models were used to create a yes/no IBPT recommendation by calculating a severity-weighted benefit-to-harm ratio; (3) Analysis outputs were summarized in a decision support tool. Based on the individual benefit-to-harm ratios, 62 and 84% of the SPRINT and CHS populations, respectively, would theoretically be recommended IBPT. The original SPRINT trial results of significant decrease in cardiovascular outcomes following IBPT persisted only in the group that received a “yes-treatment” recommendation by the suggested process, while the rate of serious adverse events was slightly higher in the "no-treatment" recommendation group. This process can be used to translate RCT data into individualized recommendations by identifying patients for whom the treatment’s benefits outweigh the harms, while considering subjective views of perceived severity of the different outcomes. The proposed approach emphasizes clinical practicality by mimicking physicians’ clinical decision-making process and integrating all recommendation outputs into a usable decision support tool.

Physician Judgement vs Model-Predicted Prognosis in Patients With Heart Failure

BACKGROUND

Previous evidence suggests that cardiologists and family doctors have limited accuracy in predicting patient prognosis. Predictive models with satisfactory accuracy for estimating mortality in patients with heart failure (HF) exist; physicians, however, seldom use these models. We evaluated the relative accuracy of physician vs model prediction to estimate 1-year survival in ambulatory patients with HF.

METHODS

We conducted a single-centre cross-sectional study involving 150 consecutive ambulatory patients with HF >18 years of age with a left ventricular ejection fraction ≤40%. Each patient's cardiologist and family doctor provided their predicted 1-year survival, and predicted survival scores were calculated using 3 models: HF Meta-Score, Seattle Heart Failure Model (SHFM), and Meta-Analysis Global Group in Chronic HF (MAGGIC) score. We compared accuracy between physician and model predictions using intraclass correlation (ICC).

RESULTS

Median predicted survival by HF cardiologists was lower (median 80%, interquartile range [IQR]: 61%-90%) than that predicted by family physicians (median 90%, IQR 70%-99%, P = 0.08). One-year median survival calculated by the HF Meta-Score (94.6%), SHFM (95.4%), and MAGGIC (88.9%,) proved as high or higher than physician estimates. Agreement among HF cardiologists (ICC 0.28-0.41) and family physicians (ICC 0.43-0.47) when compared with 1-year model-predicted survival scores proved limited, whereas the 3 models agreed well (ICC > 0.65).

CONCLUSIONS

HF cardiologists underestimated survival in comparison with family physicians, whereas both physician estimates were lower than calculated model estimates. Our results provide additional evidence of potential inaccuracy of physician survival predictions in ambulatory patients with HF. These results should be validated in longitudinal studies collecting actual survival.

Personalized medicine for patients with COPD: where are we?

Chronic airflow limitation is the common denominator of patients with chronic obstructive pulmonary disease (COPD). However, it is not possible to predict morbidity and mortality of individual patients based on the degree of lung function impairment, nor does the degree of airflow limitation allow guidance regarding therapies. Over the last decades, understanding of the factors contributing to the heterogeneity of disease trajectories, clinical presentation, and response to existing therapies has greatly advanced. Indeed, diagnostic assessment and treatment algorithms for COPD have become more personalized. In addition to the pulmonary abnormalities and inhaler therapies, extra-pulmonary features and comorbidities have been studied and are considered essential components of comprehensive disease management, including lifestyle interventions. Despite these advances, predicting and/or modifying the course of the disease remains currently impossible, and selection of patients with a beneficial response to specific interventions is unsatisfactory. Consequently, non-response to pharmacologic and non-pharmacologic treatments is common, and many patients have refractory symptoms. Thus, there is an ongoing urgency for a more targeted and holistic management of the disease, incorporating the basic principles of P4 medicine (predictive, preventive, personalized, and participatory). This review describes the current status and unmet needs regarding personalized medicine for patients with COPD. Also, it proposes a systems medicine approach, integrating genetic, environmental, (micro)biological, and clinical factors in experimental and computational models in order to decipher the multilevel complexity of COPD. Ultimately, the acquired insights will enable the development of clinical decision support systems and advance personalized medicine for patients with COPD.

Exploring the role of teams and technology in patients' medication decision making

OBJECTIVES

We know little about how electronic health records (EHRs) should be designed to help patients, pharmacists, and physicians participate in interprofessional shared decision making. We used a qualitative approach to understand better how patients make decisions with their health care team, how this information influences decision making about their medications, and finally, how this process can be improved through the use of EHRs.

DESIGN

Participants from 4 regions across Canada took part in a semistructured interview and completed a brief demographic survey. The interview transcripts were thematically analyzed by means of the multidisciplinary framework method.

SETTINGS AND PARTICIPANTS

Thirty participants, 18 years of age and older with at least one chronic illness, were recruited from across Canada. We interviewed participants in their homes, at the school of pharmacy, or another location of their choosing.

RESULTS

We identified 4 main themes: (1) complexity of patient decision making: who, where, what, when, why; (2) relationships with physicians and pharmacists: who do I trust for what?; (3) accessing health information for decision making: how much and from where?; and (4) patients' methods of managing information for health decision making. Across the themes, participants appreciated expert advice from professionals and wanted to be informed about all options, despite concerns about limited knowledge. EHRs were perceived as a potential solution to many of the barriers identified.

CONCLUSION

Patients make decisions with their health care providers as well as with family and friends. The pharmacist and physicians play different roles in helping patients in making decisions. We found that making EHRs accessible not only to health care providers but also to patients can provide a cohesive and clear context for making medication-related decisions. EHRs may facilitate clear communication, foster interprofessional understanding, and improve patient access to their health information. Future research should examine how to develop EHRs that are adaptive to user needs and desires.

THE HIGH COST OF LOW VALUE CARE

The main focus of this study is bridging the "evidence gap" between frontline decision-making in health care and the actual evidence, with the hope of reducing unnecessary diagnostic testing and treatments. From our work in pulmonary embolism (PE) and over ordering of computed tomography pulmonary angiography, we integrated the highly validated Wells' criteria into the electronic health record at two of our major academic tertiary hospitals. The Wells' clinical decision support tool triggered for patients being evaluated for PE and therefore determined a patients' pretest probability for having a PE. There were 12,759 patient visits representing 11,836 patients, 51% had no D-dimer, 41% had a negative D-dimer, and 9% had a positive D-dimer. Our study gave us an opportunity to determine which patients were very low probabilities for PE, with no need for further testing.

Applying Artificial Intelligence to Address the Knowledge Gaps in Cancer Care

BACKGROUND

Rapid advances in science challenge the timely adoption of evidence-based care in community settings. To bridge the gap between what is possible and what is practiced, we researched approaches to developing an artificial intelligence (AI) application that can provide real-time patient-specific decision support.

MATERIALS AND METHODS

The Oncology Expert Advisor (OEA) was designed to simulate peer-to-peer consultation with three core functions: patient history summarization, treatment options recommendation, and management advisory. Machine-learning algorithms were trained to construct a dynamic summary of patients cancer history and to suggest approved therapy or investigative trial options. All patient data used were retrospectively accrued. Ground truth was established for approximately 1,000 unique patients. The full Medline database of more than 23 million published abstracts was used as the literature corpus.

RESULTS

OEA's accuracies of searching disparate sources within electronic medical records to extract complex clinical concepts from unstructured text documents varied, with F1 scores of 90%-96% for non-time-dependent concepts (e.g., diagnosis) and F1 scores of 63%-65% for time-dependent concepts (e.g., therapy history timeline). Based on constructed patient profiles, OEA suggests approved therapy options linked to supporting evidence (99.9% recall; 88% precision), and screens for eligible clinical trials on ClinicalTrials.gov (97.9% recall; 96.9% precision).

CONCLUSION

Our results demonstrated technical feasibility of an AI-powered application to construct longitudinal patient profiles in context and to suggest evidence-based treatment and trial options. Our experience highlighted the necessity of collaboration across clinical and AI domains, and the requirement of clinical expertise throughout the process, from design to training to testing.

IMPLICATIONS FOR PRACTICE

Artificial intelligence (AI)-powered digital advisors such as the Oncology Expert Advisor have the potential to augment the capacity and update the knowledge base of practicing oncologists. By constructing dynamic patient profiles from disparate data sources and organizing and vetting vast literature for relevance to a specific patient, such AI applications could empower oncologists to consider all therapy options based on the latest scientific evidence for their patients, and help them spend less time on information "hunting and gathering" and more time with the patients. However, realization of this will require not only AI technology maturation but also active participation and leadership by clincial experts.

Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse

BACKGROUND

Enormous amounts of data are recorded routinely in health care as part of the care process, primarily for managing individual patient care. There are significant opportunities to use these data for other purposes, many of which would contribute to establishing a learning health system. This is particularly true for data recorded in primary care settings, as in many countries, these are the first place patients turn to for most health problems.

OBJECTIVE

In this paper, we discuss whether data that are recorded routinely as part of the health care process in primary care are actually fit to use for other purposes such as research and quality of health care indicators, how the original purpose may affect the extent to which the data are fit for another purpose, and the mechanisms behind these effects. In doing so, we want to identify possible sources of bias that are relevant for the use and reuse of these type of data.

METHODS

This paper is based on the authors' experience as users of electronic health records data, as general practitioners, health informatics experts, and health services researchers. It is a product of the discussions they had during the Translational Research and Patient Safety in Europe (TRANSFoRm) project, which was funded by the European Commission and sought to develop, pilot, and evaluate a core information architecture for the learning health system in Europe, based on primary care electronic health records.

RESULTS

We first describe the different stages in the processing of electronic health record data, as well as the different purposes for which these data are used. Given the different data processing steps and purposes, we then discuss the possible mechanisms for each individual data processing step that can generate biased outcomes. We identified 13 possible sources of bias. Four of them are related to the organization of a health care system, whereas some are of a more technical nature.

CONCLUSIONS

There are a substantial number of possible sources of bias; very little is known about the size and direction of their impact. However, anyone that uses or reuses data that were recorded as part of the health care process (such as researchers and clinicians) should be aware of the associated data collection process and environmental influences that can affect the quality of the data. Our stepwise, actor- and purpose-oriented approach may help to identify these possible sources of bias. Unless data quality issues are better understood and unless adequate controls are embedded throughout the data lifecycle, data-driven health care will not live up to its expectations. We need a data quality research agenda to devise the appropriate instruments needed to assess the magnitude of each of the possible sources of bias, and then start measuring their impact. The possible sources of bias described in this paper serve as a starting point for this research agenda.

Automating Quality Measures for Heart Failure Using Natural Language Processing: A Descriptive Study in the Department of Veterans Affairs

Background

We developed an accurate, stakeholder-informed, automated, natural language processing (NLP) system to measure the quality of heart failure (HF) inpatient care, and explored the potential for adoption of this system within an integrated health care system.

Objective

To accurately automate a United States Department of Veterans Affairs (VA) quality measure for inpatients with HF.

Methods

We automated the HF quality measure Congestive Heart Failure Inpatient Measure 19 (CHI19) that identifies whether a given patient has left ventricular ejection fraction (LVEF) <40%, and if so, whether an angiotensin-converting enzyme inhibitor or angiotensin-receptor blocker was prescribed at discharge if there were no contraindications. We used documents from 1083 unique inpatients from eight VA medical centers to develop a reference standard (RS) to train (n=314) and test (n=769) the Congestive Heart Failure Information Extraction Framework (CHIEF). We also conducted semi-structured interviews (n=15) for stakeholder feedback on implementation of the CHIEF.

Results

The CHIEF classified each hospitalization in the test set with a sensitivity (SN) of 98.9% and positive predictive value of 98.7%, compared with an RS and SN of 98.5% for available External Peer Review Program assessments. Of the 1083 patients available for the NLP system, the CHIEF evaluated and classified 100% of cases. Stakeholders identified potential implementation facilitators and clinical uses of the CHIEF.

Conclusions

The CHIEF provided complete data for all patients in the cohort and could potentially improve the efficiency, timeliness, and utility of HF quality measurements.

The Value of Electronic Medical Record Implementation in Mental Health Care: A Case Study

Background

Electronic medical records (EMR) have been implemented in many organizations to improve the quality of care. Evidence supporting the value added to a recovery-oriented mental health facility is lacking.

Objective

The goal of this project was to implement and customize a fully integrated EMR system in a specialized, recovery-oriented mental health care facility. This evaluation examined the outcomes of quality improvement initiatives driven by the EMR to determine the value that the EMR brought to the organization.

Methods

The setting was a tertiary-level mental health facility in Ontario, Canada. Clinical informatics and decision support worked closely with point-of-care staff to develop workflows and documentation tools in the EMR. The primary initiatives were implementation of modules for closed loop medication administration, collaborative plan of care, clinical practice guidelines for schizophrenia, restraint minimization, the infection prevention and control surveillance status board, drug of abuse screening, and business intelligence.

Results

Medication and patient scan rates have been greater than 95% since April 2014, mitigating the adverse effects of medication errors. Specifically, between April 2014 and March 2015, only 1 moderately severe and 0 severe adverse drug events occurred. The number of restraint incidents decreased 19.7%, which resulted in cost savings of more than Can $1.4 million (US $1.0 million) over 2 years. Implementation of clinical practice guidelines for schizophrenia increased adherence to evidence-based practices, standardizing care across the facility. Improved infection prevention and control surveillance reduced the number of outbreak days from 47 in the year preceding implementation of the status board to 7 days in the year following. Decision support to encourage preferential use of the cost-effective drug of abuse screen when clinically indicated resulted in organizational cost savings.

Conclusions

EMR implementation allowed Ontario Shores Centre for Mental Health Sciences to use data analytics to identify and select appropriate quality improvement initiatives, supporting patient-centered, recovery-oriented practices and providing value at the clinical, organizational, and societal levels.

Avoiding alert fatigue in pulmonary embolism decision support: a new method to examine 'trigger rates'

A clinical decision support system (CDSS) is integrated into the electronic health record (EHR) and allows physicians to easily use a clinical decision support (CDS) tool. However, often CDSSs are integrated into the EHR with poor adoption rates. One reason for this is secondary to 'trigger fatigue'. Therefore, we developed a new and innovative usability process named 'sensitivity and specificity trigger analysis' (SSTA) as part of our larger project around a pulmonary embolism decision support tool. SSTA will enable programmers to examine optimal trigger rates prior to the integration of a CDS tool into the EHR, by using a formal method of analysis. We performed a retrospective chart review. The outcome of interest was physician ordering of a CT angiography (CTA). Phrases that signify common symptoms associated with pulmonary embolism were assessed as possible triggers for the CDSS tool. We then analysed each trigger's ability to predict physician ordering of a CTA. We found that the most sensitive way to trigger the Pulmonary Embolism CDS tool while still maintaining a high specificity was by combining 1 or more pertinent symptoms with 1 or more elements of the Wells criteria. This study explored a unique methodology, SSTA, used to limit inaccurate triggering of a CDS tool prior to integration into the EHR. This methodology can be applied to other studies aiming to decrease triggering rates and increase adoption rates of previously validated CDSS tools.