Accuracy of using automated methods for detecting adverse events from electronic health record data: a research protocol

Automated Detection of Patient Harm: Implementation and Prospective Evaluation of a Real-Time Broad-Spectrum Surveillance Application in a Hospital With Limited Resources

OBJECTIVES

This study aimed to prospectively validate an application that automates the detection of broad categories of hospital adverse events (AEs) extracted from a basic hospital information system, and to efficiently mobilize resources to reduce the level of acquired patient harm.

METHODS

Data were collected from an internally designed software, extracting results from 14 triggers indicative of patient harm, querying clinical and administrative databases including all inpatient admissions (n = 8760) from October 2019 to June 2020. Representative samples of the triggered cases were clinically validated using chart review by a consensus expert panel. The positive predictive value (PPV) of each trigger was evaluated, and the detection sensitivity of the surveillance system was estimated relative to incidence ranges in the literature.

RESULTS

The system identified 394 AEs among 946 triggered cases, associated with 291 patients, yielding an overall PPV of 42%. Variability was observed among the trigger PPVs and among the estimated detection sensitivities across the harm categories, the highest being for the healthcare-associated infections. The median length of stay of patients with an AE showed to be significantly higher than the median for the overall patient population.

CONCLUSIONS

This application was able to identify AEs across a broad spectrum of harm categories, in a real-time manner, while reducing the use of resources required by other harm detection methods. Such a system could serve as a promising patient safety tool for AE surveillance, allowing for timely, targeted, and resource-efficient interventions, even for hospitals with limited resources.

medExtractR: A targeted, customizable approach to medication extraction from electronic health records

OBJECTIVE

We developed medExtractR, a natural language processing system to extract medication information from clinical notes. Using a targeted approach, medExtractR focuses on individual drugs to facilitate creation of medication-specific research datasets from electronic health records.

MATERIALS AND METHODS

Written using the R programming language, medExtractR combines lexicon dictionaries and regular expressions to identify relevant medication entities (eg, drug name, strength, frequency). MedExtractR was developed on notes from Vanderbilt University Medical Center, using medications prescribed with varying complexity. We evaluated medExtractR and compared it with 3 existing systems: MedEx, MedXN, and CLAMP (Clinical Language Annotation, Modeling, and Processing). We also demonstrated how medExtractR can be easily tuned for better performance on an outside dataset using the MIMIC-III (Medical Information Mart for Intensive Care III) database.

RESULTS

On 50 test notes per development drug and 110 test notes for an additional drug, medExtractR achieved high overall performance (F-measures >0.95), exceeding performance of the 3 existing systems across all drugs. MedExtractR achieved the highest F-measure for each individual entity, except drug name and dose amount for allopurinol. With tuning and customization, medExtractR achieved F-measures >0.90 in the MIMIC-III dataset.

DISCUSSION

The medExtractR system successfully extracted entities for medications of interest. High performance in entity-level extraction provides a strong foundation for developing robust research datasets for pharmacological research. When working with new datasets, medExtractR should be tuned on a small sample of notes before being broadly applied.

CONCLUSIONS

The medExtractR system achieved high performance extracting specific medications from clinical text, leading to higher-quality research datasets for drug-related studies than some existing general-purpose medication extraction tools.

Information Extraction from Electronic Medical Records Using Multitask Recurrent Neural Network with Contextual Word Embedding

Clinical named entity recognition is an essential task for humans to analyze large-scale electronic medical records efficiently. Traditional rule-based solutions need considerable human effort to build rules and dictionaries; machine learning-based solutions need laborious feature engineering. For the moment, deep learning solutions like Long Short-term Memory with Conditional Random Field (LSTM–CRF) achieved considerable performance in many datasets. In this paper, we developed a multitask attention-based bidirectional LSTM–CRF (Att-biLSTM–CRF) model with pretrained Embeddings from Language Models (ELMo) in order to achieve better performance. In the multitask system, an additional task named entity discovery was designed to enhance the model’s perception of unknown entities. Experiments were conducted on the 2010 Informatics for Integrating Biology & the Bedside/Veterans Affairs (I2B2/VA) dataset. Experimental results show that our model outperforms the state-of-the-art solution both on the single model and ensemble model. Our work proposes an approach to improve the recall in the clinical named entity recognition task based on the multitask mechanism.

Making sense of abbreviations in nursing notes: A case study on mortality prediction

Unstructured data from electronic health records hold potential for improving predictive models for health outcomes. Efforts to extract structured information from the unstructured data used text mining methodologies, such as topic modeling and sentiment analysis. However, such methods do not account for abbreviations. Nursing notes have valuable information about nurses' assessments and interventions, and the abbreviation use is common. Thus, abbreviation disambiguation may add more insight when using unstructured text for predictive modeling. We present a new process to extract structured information from nursing notes through abbreviation normalization, lemmatization, and stop word removal. Our study found that abbreviation disambiguation in nursing notes for subsequent topic modeling and sentiment analysis improved prediction of in-hospital and 30-day mortality while controlling for comorbidity.

Automated safety event monitoring using electronic medical records in a clinical trial setting: Validation study using the VA NEPHRON-D trial

Background/Aims:

Electronic medical records are now frequently used for capturing patient-level data in clinical trials. Within the Veterans Affairs health care system, electronic medical record data have been widely used in clinical trials to assess eligibility, facilitate referrals for recruitment, and conduct follow-up and safety monitoring. Despite the potential for increased efficiency in using electronic medical records to capture safety data via a centralized algorithm, it is important to evaluate the integrity and accuracy of electronic medical record–captured data. To this end, this investigation assesses data collection, both for general and study-specific safety endpoints, by comparing electronic medical record–based safety monitoring versus safety data collected during the course of the Veterans Affairs Nephropathy in Diabetes (VA NEPHRON-D) clinical trial.

Methods:

The VA NEPHRON-D study was a multicenter, double-blind, randomized clinical trial designed to compare the effect of combination therapy (losartan plus lisinopril) versus monotherapy (losartan) on the progression of kidney disease in individuals with diabetes and proteinuria. The trial’s safety outcomes included serious adverse events, hyperkalemia, and acute kidney injury. A subset of the participants (~62%, n = 895) enrolled in the trial’s long-term follow-up sub-study and consented to electronic medical record data collection. We applied an automated algorithm to search and capture safety data using the VA Corporate Data Warehouse which houses electronic medical record data. Using study safety data reported during the trial as the gold standard, we evaluated the sensitivity and precision of electronic medical record–based safety data and related treatment effects.

Results:

The sensitivity of the electronic medical record–based safety for hospitalizations was 65.3% without non-VA hospitalization events and 92.3% with the non-VA hospitalization events included. The sensitivity was only 54.3% for acute kidney injury and 87.3% for hyperkalemia. The precision of electronic medical record–based safety data was 89.4%, 38%, and 63.2% for hospitalization, acute kidney injury, and hyperkalemia, respectively. Relative treatment differences under the study and electronic medical record settings were 15% and 3% for hospitalization, 123% and 29% for acute kidney injury, and 238% and 140% for hyperkalemia, respectively.

Conclusion:

The accuracy of using automated electronic medical record safety data depends on the events of interest. Identification of all-cause hospitalizations would be reliable if search methods could, in addition to VA hospitalizations, also capture non-VA hospitalizations. However, hospitalization is different from a cause-specific serious adverse event that could be more sensitive to treatment effects. In addition, some study-specific safety events were not easily identified using the electronic medical records. This limits the effectiveness of the automated central database search for purposes of safety monitoring. Hence, this data captured approach should be carefully considered when implementing endpoint data collection in future pragmatic trials.

A method to identify pediatric high-risk diagnoses missed in the emergency department

BACKGROUND

Diagnostic error can lead to increased morbidity, mortality, healthcare utilization and cost. The 2015 National Academy of Medicine report "Improving Diagnosis in Healthcare" called for improving diagnostic accuracy by developing innovative electronic approaches to reduce medical errors, including missed or delayed diagnosis. The objective of this article was to develop a process to detect potential diagnostic discrepancy between pediatric emergency and inpatient discharge diagnosis using a computer-based tool facilitating expert review.

METHODS

Using a literature search and expert opinion, we identified 10 pediatric diagnoses with potential for serious consequences if missed or delayed. We then developed and applied a computerized tool to identify linked emergency department (ED) encounters and hospitalizations with these discharge diagnoses. The tool identified discordance between ED and hospital discharge diagnoses. Cases identified as discordant were manually reviewed by pediatric emergency medicine experts to confirm discordance.

RESULTS

Our computerized tool identified 55,233 ED encounters for hospitalized children over a 5-year period, of which 2161 (3.9%) had one of the 10 selected high-risk diagnoses. After expert record review, we identified 67 (3.1%) cases with discordance between ED and hospital discharge diagnoses. The most common discordant diagnoses were Kawasaki disease and pancreatitis.

CONCLUSIONS

We successfully developed and applied a semi-automated process to screen a large volume of hospital encounters to identify discordant diagnoses for selected pediatric medical conditions. This process may be valuable for informing and improving ED diagnostic accuracy.

Associations between nurse education and experience and the risk of mortality and adverse events in acute care hospitals: A systematic review of observational studies

OBJECTIVES

To provide knowledge from the summarization of the evidence on the: a) associations between nurse education and experience and the occurrence of mortality and adverse events in acute care hospitals, and; b) benefits to patients and organizations of the recent Institute of Medicine's recommendation that 80% of registered nurses should be educated at the baccalaureate degree by 2020.

DATA SOURCES

A systematic search of English and French literature was conducted in six electronic databases: 1) Medline, 2) PubMed, 3) CINAHL, 4) Scopus, 5) Campbell, and 6) Cochrane databases. Additional studies were identified by searching bibliographies, prior reviews, and by contacting authors.

REVIEW METHOD

Studies were included if they: a) were published between January 1996 and August 2017; b) were based on a quantitative research design; c) examined the associations between registered nurse education or experience and at least one independently measured adverse event, and; d) were conducted in an adult acute care setting. Data were independently extracted, analysed, and synthesized by two authors and discrepancies were resolved by consensus. The methodological heterogeneity of the reviewed studies precluded the use of meta-analysis techniques. However, the methodological quality of each study was assessed using the STROBE criteria.

FINDINGS

Among 2109 retrieved articles, 27 studies (24 cross-sectional and three longitudinal studies) met our inclusion criteria. These studies examined 18 distinct adverse events, with mortality and failure to rescue being the most frequently investigated events. Overall, higher levels of education were associated with lower risks of failure to rescue and mortality in 75% and 61.1% of the reviewed studies pertaining to these adverse events, respectively. Nurse education was inconsistently related to the occurrence of the other events, which were the focus of only a small number of studies. Only one study examined the 80% threshold proposed by the Institute of Medicine and found evidence that it is associated with lower odds of hospital readmission and shorter lengths of stay, but unrelated to mortality. Nurse experience was inconsistently related to adverse event occurrence.

CONCLUSION

While evidence suggests that higher nurse education is associated with lower risks of mortality and failure to rescue, longitudinal studies are needed to better ascertain these associations and determine the specific thresholds that minimize risks. Further studies are needed to better document the association of nurse education and experience with other nursing-sensitive adverse events, as well as the benefits to patients and organizations of the Institute of Medicine's recommendation.

Electronic surveillance and using administrative data to identify healthcare associated infections

PURPOSE OF REVIEW

Traditional surveillance of healthcare associated infections (HCAI) is time consuming and error-prone. We have analysed literature of the past year to look at new developments in this field. It is divided into three parts: new algorithms for electronic surveillance, the use of administrative data for surveillance of HCAI, and the definition of new endpoints of surveillance, in accordance with an automatic surveillance approach.

RECENT FINDINGS

Most studies investigating electronic surveillance of HCAI have concentrated on bloodstream infection or surgical site infection. However, the lack of important parameters in hospital databases can lead to misleading results. The accuracy of administrative coding data was poor at identifying HCAI. New endpoints should be defined for automatic detection, with the most crucial step being to win clinicians' acceptance.

SUMMARY

Electronic surveillance with conventional endpoints is a successful method when hospital information systems implemented key changes and enhancements. One requirement is the access to systems for hospital administration and clinical databases.Although the primary source of data for HCAI surveillance is not administrative coding data, these are important components of a hospital-wide programme of automated surveillance. The implementation of new endpoints for surveillance is an approach which needs to be discussed further.

Clinical information extraction applications: A literature review

BACKGROUND

With the rapid adoption of electronic health records (EHRs), it is desirable to harvest information and knowledge from EHRs to support automated systems at the point of care and to enable secondary use of EHRs for clinical and translational research. One critical component used to facilitate the secondary use of EHR data is the information extraction (IE) task, which automatically extracts and encodes clinical information from text.

OBJECTIVES

In this literature review, we present a review of recent published research on clinical information extraction (IE) applications.

METHODS

A literature search was conducted for articles published from January 2009 to September 2016 based on Ovid MEDLINE In-Process & Other Non-Indexed Citations, Ovid MEDLINE, Ovid EMBASE, Scopus, Web of Science, and ACM Digital Library.

RESULTS

A total of 1917 publications were identified for title and abstract screening. Of these publications, 263 articles were selected and discussed in this review in terms of publication venues and data sources, clinical IE tools, methods, and applications in the areas of disease- and drug-related studies, and clinical workflow optimizations.

CONCLUSIONS

Clinical IE has been used for a wide range of applications, however, there is a considerable gap between clinical studies using EHR data and studies using clinical IE. This study enabled us to gain a more concrete understanding of the gap and to provide potential solutions to bridge this gap.

Automated Screening of Emergency Department Notes for Drug-Associated Bleeding Adverse Events Occurring in Older Adults

Objective To conduct research to show the value of text mining for automatically identifying suspected bleeding adverse drug events (ADEs) in the emergency department (ED). Methods A corpus of ED admission notes was manually annotated for bleeding ADEs. The notes were taken for patients ≥ 65 years of age who had an ICD-9 code for bleeding, the presence of hemoglobin value ≤ 8 g/dL, or were transfused > 2 units of packed red blood cells. This training corpus was used to develop bleeding ADE algorithms using Random Forest and Classification and Regression Tree (CART). A completely separate set of notes was annotated and used to test the classification performance of the final models using the area under the ROC curve (AUROC). Results The best performing CART resulted in an AUROC on the training set of 0.882. The model's AUROC on the test set was 0.827. At a sensitivity of 0.679, the model had a specificity of 0.908 and a positive predictive value (PPV) of 0.814. It had a relatively simple and intuitive structure consisting of 13 decision nodes and 14 leaf nodes. Decision path probabilities ranged from 0.041 to 1.0. The AUROC for the best performing Random Forest method on the training set was 0.917. On the test set, the model's AUROC was 0.859. At a sensitivity of 0.274, the model had a specificity of 0.986 and a PPV of 0.92. Conclusion Both models accurately identify bleeding ADEs using the presence or absence of certain clinical concepts in ED admission notes for older adult patients. The CART model is particularly noteworthy because it does not require significant technical overhead to implement. Future work should seek to replicate the results on a larger test set pulled from another institution.

Accuracy and generalizability of using automated methods for identifying adverse events from electronic health record data: a validation study protocol

Background

Adverse events (AEs) in acute care hospitals are frequent and associated with significant morbidity, mortality, and costs. Measuring AEs is necessary for quality improvement and benchmarking purposes, but current detection methods lack in accuracy, efficiency, and generalizability. The growing availability of electronic health records (EHR) and the development of natural language processing techniques for encoding narrative data offer an opportunity to develop potentially better methods. The purpose of this study is to determine the accuracy and generalizability of using automated methods for detecting three high-incidence and high-impact AEs from EHR data: a) hospital-acquired pneumonia, b) ventilator-associated event and, c) central line-associated bloodstream infection.

Methods

This validation study will be conducted among medical, surgical and ICU patients admitted between 2013 and 2016 to the Centre hospitalier universitaire de Sherbrooke (CHUS) and the McGill University Health Centre (MUHC), which has both French and English sites. A random 60% sample of CHUS patients will be used for model development purposes (cohort 1, development set). Using a random sample of these patients, a reference standard assessment of their medical chart will be performed. Multivariate logistic regression and the area under the curve (AUC) will be employed to iteratively develop and optimize three automated AE detection models (i.e., one per AE of interest) using EHR data from the CHUS. These models will then be validated on a random sample of the remaining 40% of CHUS patients (cohort 1, internal validation set) using chart review to assess accuracy. The most accurate models developed and validated at the CHUS will then be applied to EHR data from a random sample of patients admitted to the MUHC French site (cohort 2) and English site (cohort 3)—a critical requirement given the use of narrative data –, and accuracy will be assessed using chart review. Generalizability will be determined by comparing AUCs from cohorts 2 and 3 to those from cohort 1.

Discussion

This study will likely produce more accurate and efficient measures of AEs. These measures could be used to assess the incidence rates of AEs, evaluate the success of preventive interventions, or benchmark performance across hospitals.

Asphyxia in the Newborn: Evaluating the Accuracy of ICD Coding, Clinical Diagnosis and Reimbursement: Observational Study at a Swiss Tertiary Care Center on Routinely Collected Health Data from 2012-2015

BACKGROUND

The ICD-10 categories of the diagnosis "perinatal asphyxia" are defined by clinical signs and a 1-minute Apgar score value. However, the modern conception is more complex and considers metabolic values related to the clinical state. A lack of consistency between the former clinical and the latter encoded diagnosis poses questions over the validity of the data. Our aim was to establish a refined classification which is able to distinctly separate cases according to clinical criteria and financial resource consumption. The hypothesis of the study is that outdated ICD-10 definitions result in differences between the encoded diagnosis asphyxia and the medical diagnosis referring to the clinical context.

METHODS

Routinely collected health data (encoding and financial data) of the University Hospital of Bern were used. The study population was chosen by selected ICD codes, the encoded and the clinical diagnosis were analyzed and each case was reevaluated. The new method categorizes the diagnoses of perinatal asphyxia into the following groups: mild, moderate and severe asphyxia, metabolic acidosis and normal clinical findings. The differences of total costs per case were determined by using one-way analysis of variance.

RESULTS

The study population included 622 cases (P20 "intrauterine hypoxia" 399, P21 "birth asphyxia" 233). By applying the new method, the diagnosis asphyxia could be ruled out with a high probability in 47% of cases and the variance of case related costs (one-way ANOVA: F (5, 616) = 55.84, p < 0.001, multiple R-squared = 0.312, p < 0.001) could be best explained. The classification of the severity of asphyxia could clearly be linked to the complexity of cases.

CONCLUSION

The refined coding method provides clearly defined diagnoses groups and has the strongest effect on the distribution of costs. It improves the diagnosis accuracy of perinatal asphyxia concerning clinical practice, research and reimbursement.