1
|
Rochefort CM, Buckeridge DL, Forster AJ. Accuracy of using automated methods for detecting adverse events from electronic health record data: a research protocol. Implement Sci 2015; 10:5. [PMID: 25567422 PMCID: PMC4296680 DOI: 10.1186/s13012-014-0197-6] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2014] [Accepted: 12/18/2014] [Indexed: 12/13/2022] Open
Abstract
Background Adverse events are associated with significant morbidity, mortality and cost in hospitalized patients. Measuring adverse events is necessary for quality improvement, but current detection methods are inaccurate, untimely and expensive. The advent of electronic health records and the development of automated methods for encoding and classifying electronic narrative data, such as natural language processing, offer an opportunity to identify potentially better methods. The objective of this study is to determine the accuracy of using automated methods for detecting three highly prevalent adverse events: a) hospital-acquired pneumonia, b) catheter-associated bloodstream infections, and c) in-hospital falls. Methods/design This validation study will be conducted at two large Canadian academic health centres: the McGill University Health Centre (MUHC) and The Ottawa Hospital (TOH). The study population consists of all medical, surgical and intensive care unit patients admitted to these centres between 2008 and 2014. An automated detection algorithm will be developed and validated for each of the three adverse events using electronic data extracted from multiple clinical databases. A random sample of MUHC patients will be used to develop the automated detection algorithms (cohort 1, development set). The accuracy of these algorithms will be assessed using chart review as the reference standard. Then, receiver operating characteristic curves will be used to identify optimal cut points for each of the data sources. Multivariate logistic regression and the areas under curve (AUC) will be used to identify the optimal combination of data sources that maximize the accuracy of adverse event detection. The most accurate algorithms will then be validated on a second random sample of MUHC patients (cohort 1, validation set), and accuracy will be measured using chart review as the reference standard. The most accurate algorithms validated at the MUHC will then be applied to TOH data (cohort 2), and their accuracy will be assessed using a reference standard assessment of the medical chart. Discussion There is a need for more accurate, timely and efficient measures of adverse events in acute care hospitals. This is a critical requirement for evaluating the effectiveness of preventive interventions and for tracking progress in patient safety through time.
Collapse
Affiliation(s)
- Christian M Rochefort
- Ingram School of Nursing, Faculty of Medicine, McGill University, Wilson Hall, 3506 University Street, Montreal, QC, H3A 2A7, Canada. .,McGill Clinical and Health Informatics Research Group, McGill University, 1140, Pine Avenue West, Montreal, QC, H3A 1A3, Canada. .,Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Purvis Hall, 1020 Pine Avenue West, Montreal, QC, H3A 1A2, Canada.
| | - David L Buckeridge
- McGill Clinical and Health Informatics Research Group, McGill University, 1140, Pine Avenue West, Montreal, QC, H3A 1A3, Canada. .,Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Purvis Hall, 1020 Pine Avenue West, Montreal, QC, H3A 1A2, Canada.
| | - Alan J Forster
- Ottawa Hospital Research Institute, Ottawa, ON, Canada. .,The Ottawa Hospital, 725 Parkdale Ave, Ottawa, ON, K1Y 4E9, Canada.
| |
Collapse
|
2
|
Rochefort CM, Verma AD, Eguale T, Lee TC, Buckeridge DL. A novel method of adverse event detection can accurately identify venous thromboembolisms (VTEs) from narrative electronic health record data. J Am Med Inform Assoc 2014; 22:155-65. [PMID: 25332356 PMCID: PMC4433368 DOI: 10.1136/amiajnl-2014-002768] [Citation(s) in RCA: 52] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
BACKGROUND Venous thromboembolisms (VTEs), which include deep vein thrombosis (DVT) and pulmonary embolism (PE), are associated with significant mortality, morbidity, and cost in hospitalized patients. To evaluate the success of preventive measures, accurate and efficient methods for monitoring VTE rates are needed. Therefore, we sought to determine the accuracy of statistical natural language processing (NLP) for identifying DVT and PE from electronic health record data. METHODS We randomly sampled 2000 narrative radiology reports from patients with a suspected DVT/PE in Montreal (Canada) between 2008 and 2012. We manually identified DVT/PE within each report, which served as our reference standard. Using a bag-of-words approach, we trained 10 alternative support vector machine (SVM) models predicting DVT, and 10 predicting PE. SVM training and testing was performed with nested 10-fold cross-validation, and the average accuracy of each model was measured and compared. RESULTS On manual review, 324 (16.2%) reports were DVT-positive and 154 (7.7%) were PE-positive. The best DVT model achieved an average sensitivity of 0.80 (95% CI 0.76 to 0.85), specificity of 0.98 (98% CI 0.97 to 0.99), positive predictive value (PPV) of 0.89 (95% CI 0.85 to 0.93), and an area under the curve (AUC) of 0.98 (95% CI 0.97 to 0.99). The best PE model achieved sensitivity of 0.79 (95% CI 0.73 to 0.85), specificity of 0.99 (95% CI 0.98 to 0.99), PPV of 0.84 (95% CI 0.75 to 0.92), and AUC of 0.99 (95% CI 0.98 to 1.00). CONCLUSIONS Statistical NLP can accurately identify VTE from narrative radiology reports.
Collapse
Affiliation(s)
- Christian M Rochefort
- Faculty of Medicine, Ingram School of Nursing, McGill University, Montreal, Canada McGill Clinical and Health Informatics Research Group, McGill University, Montreal, Canada Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Montreal, Canada
| | - Aman D Verma
- McGill Clinical and Health Informatics Research Group, McGill University, Montreal, Canada Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Montreal, Canada
| | - Tewodros Eguale
- McGill Clinical and Health Informatics Research Group, McGill University, Montreal, Canada Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Todd C Lee
- McGill University Health Centre (MUHC), Montreal, Canada
| | - David L Buckeridge
- McGill Clinical and Health Informatics Research Group, McGill University, Montreal, Canada Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Montreal, Canada
| |
Collapse
|