1
|
Zitu MM, Zhang S, Owen DH, Chiang C, Li L. Generalizability of machine learning methods in detecting adverse drug events from clinical narratives in electronic medical records. Front Pharmacol 2023; 14:1218679. [PMID: 37502211 PMCID: PMC10368879 DOI: 10.3389/fphar.2023.1218679] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Accepted: 06/26/2023] [Indexed: 07/29/2023] Open
Abstract
We assessed the generalizability of machine learning methods using natural language processing (NLP) techniques to detect adverse drug events (ADEs) from clinical narratives in electronic medical records (EMRs). We constructed a new corpus correlating drugs with adverse drug events using 1,394 clinical notes of 47 randomly selected patients who received immune checkpoint inhibitors (ICIs) from 2011 to 2018 at The Ohio State University James Cancer Hospital, annotating 189 drug-ADE relations in single sentences within the medical records. We also used data from Harvard's publicly available 2018 National Clinical Challenge (n2c2), which includes 505 discharge summaries with annotations of 1,355 single-sentence drug-ADE relations. We applied classical machine learning (support vector machine (SVM)), deep learning (convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM)), and state-of-the-art transformer-based (bidirectional encoder representations from transformers (BERT) and ClinicalBERT) methods trained and tested in the two different corpora and compared performance among them to detect drug-ADE relationships. ClinicalBERT detected drug-ADE relationships better than the other methods when trained using our dataset and tested in n2c2 (ClinicalBERT F-score, 0.78; other methods, F-scores, 0.61-0.73) and when trained using the n2c2 dataset and tested in ours (ClinicalBERT F-score, 0.74; other methods, F-scores, 0.55-0.72). Comparison among several machine learning methods demonstrated the superior performance and, therefore, the greatest generalizability of findings of ClinicalBERT for the detection of drug-ADE relations from clinical narratives in electronic medical records.
Collapse
Affiliation(s)
- Md Muntasir Zitu
- Department of Biomedical Informatics, College of Medicine, The Ohio State University, Columbus, OH, United States
| | - Shijun Zhang
- Department of Biomedical Informatics, College of Medicine, The Ohio State University, Columbus, OH, United States
| | - Dwight H. Owen
- Department of Internal Medicine, College of Medicine, The Ohio State University, Columbus, OH, United States
| | - Chienwei Chiang
- Department of Biomedical Informatics, College of Medicine, The Ohio State University, Columbus, OH, United States
| | - Lang Li
- Department of Biomedical Informatics, College of Medicine, The Ohio State University, Columbus, OH, United States
| |
Collapse
|
2
|
Gaspar F, Lutters M, Beeler PE, Lang PO, Burnand B, Rinaldi F, Lovis C, Csajka C, Le Pogam MA. Automatic Detection of Adverse Drug Events in Geriatric Care: Study Proposal. JMIR Res Protoc 2022; 11:e40456. [PMID: 36378522 PMCID: PMC9709671 DOI: 10.2196/40456] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Revised: 07/06/2022] [Accepted: 07/07/2022] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND One-third of older inpatients experience adverse drug events (ADEs), which increase their mortality, morbidity, and health care use and costs. In particular, antithrombotic drugs are among the most at-risk medications for this population. Reporting systems have been implemented at the national, regional, and provider levels to monitor ADEs and design prevention strategies. Owing to their well-known limitations, automated detection technologies based on electronic medical records (EMRs) are being developed to routinely detect or predict ADEs. OBJECTIVE This study aims to develop and validate an automated detection tool for monitoring antithrombotic-related ADEs using EMRs from 4 large Swiss hospitals. We aim to assess cumulative incidences of hemorrhages and thromboses in older inpatients associated with the prescription of antithrombotic drugs, identify triggering factors, and propose improvements for clinical practice. METHODS This project is a multicenter, cross-sectional study based on 2015 to 2016 EMR data from 4 large hospitals in Switzerland: Lausanne, Geneva, and Zürich university hospitals, and Baden Cantonal Hospital. We have included inpatients aged ≥65 years who stayed at 1 of the 4 hospitals during 2015 or 2016, received at least one antithrombotic drug during their stay, and signed or were not opposed to a general consent for participation in research. First, clinical experts selected a list of relevant antithrombotic drugs along with their side effects, risks, and confounding factors. Second, administrative, clinical, prescription, and laboratory data available in the form of free text and structured data were extracted from study participants' EMRs. Third, several automated rule-based and machine learning-based algorithms are being developed, allowing for the identification of hemorrhage and thromboembolic events and their triggering factors from the extracted information. Finally, we plan to validate the developed detection tools (one per ADE type) through manual medical record review. Performance metrics for assessing internal validity will comprise the area under the receiver operating characteristic curve, F1-score, sensitivity, specificity, and positive and negative predictive values. RESULTS After accounting for the inclusion and exclusion criteria, we will include 34,522 residents aged ≥65 years. The data will be analyzed in 2022, and the research project will run until the end of 2022 to mid-2023. CONCLUSIONS This project will allow for the introduction of measures to improve safety in prescribing antithrombotic drugs, which today remain among the drugs most involved in ADEs. The findings will be implemented in clinical practice using indicators of adverse events for risk management and training for health care professionals; the tools and methodologies developed will be disseminated for new research in this field. The increased performance of natural language processing as an important complement to structured data will bring existing tools to another level of efficiency in the detection of ADEs. Currently, such systems are unavailable in Switzerland. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/40456.
Collapse
Affiliation(s)
- Frederic Gaspar
- Center for Research and Innovation in Clinical Pharmaceutical Sciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
- School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland
- Institute of Pharmaceutical Sciences of Western Switzerland, University of Geneva, University of Lausanne, Geneva and Lausanne, Switzerland
| | - Monika Lutters
- Service of Clinical Pharmacy, Baden University Hospital, Baden, Switzerland
| | - Patrick Emanuel Beeler
- Division of Occupational and Environmental Medicine, Epidemiology, Biostatistics and Prevention Institute, University of Zurich and University Hospital Zurich, Zurich, Switzerland
| | | | - Bernard Burnand
- Unisanté Center for Primary Care and Public Health, Department of Epidemiology and Health Systems, University of Lausanne, Lausanne, Switzerland
| | - Fabio Rinaldi
- Dalle Molle Institute for Artificial Intelligence Research, Scuola Universitaria Professionale della Svizzera Italiana, Universita della Svizzera Italiana, Lugano, Switzerland
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
- Swiss Institute of Bioinformatics, Lausanne, Switzerland
- Fondazione Bruno Kessler, Trento, Italy
| | - Christian Lovis
- Division of Medical Information Sciences, Geneva University Hospitals and University of Geneva, Geneva, Switzerland
| | - Chantal Csajka
- Center for Research and Innovation in Clinical Pharmaceutical Sciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
- School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland
- Institute of Pharmaceutical Sciences of Western Switzerland, University of Geneva, University of Lausanne, Geneva and Lausanne, Switzerland
| | - Marie-Annick Le Pogam
- Unisanté Center for Primary Care and Public Health, Department of Epidemiology and Health Systems, University of Lausanne, Lausanne, Switzerland
| |
Collapse
|
3
|
Roosan D, Law AV, Roosan MR, Li Y. Artificial Intelligent Context-Aware Machine-Learning Tool to Detect Adverse Drug Events from Social Media Platforms. J Med Toxicol 2022; 18:311-320. [PMID: 36097239 PMCID: PMC9492823 DOI: 10.1007/s13181-022-00906-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2022] [Revised: 07/15/2022] [Accepted: 07/18/2022] [Indexed: 10/14/2022] Open
Abstract
INTRODUCTION Pharmacovigilance (PV) has proven to detect post-marketing adverse drug events (ADE). Previous research used the natural language processing (NLP) tool to extract unstructured texts relevant to ADEs. However, texts without context reduce the efficiency of such algorithms. Our objective was to develop and validate an innovative NLP tool, aTarantula, using a context-aware machine-learning algorithm to detect existing ADEs from social media using an aggregated lexicon. METHOD aTarantula utilized FastText embeddings and an aggregated lexicon to extract contextual data from three patient forums (i.e., MedHelp, MedsChat, and PatientInfo) taking warfarin. The lexicon used warfarin package inserts and synonyms of warfarin ADEs from UMLS and FAERS databases. Data was stored on SQLite and then refined and manually checked by three clinical pharmacists for validation. RESULTS Multiple organ systems where the most frequent ADE were reported at 1.50%, followed by CNS side effects at 1.19%. Lymphatic system ADEs were the least common side effect reported at 0.09%. The overall Spearman rank correlation coefficient between patient-reported data from the forums and FAERS was 0.19. As determined by pharmacist validation, aTarantula had a sensitivity of 84.2% and a specificity of 98%. Three clinical pharmacists manually validated our results. Finally, we created an aggregated lexicon for mining ADEs from social media. CONCLUSION We successfully developed aTarantula, a machine-learning algorithmn based on artificial intelligence to extract warfarin-related ADEs from online social discussion forums automatically. Our study shows that it is feasible to use aTarantula to detect ADEs. Future researchers can validate aTarantula on the diverse dataset.
Collapse
Affiliation(s)
- Don Roosan
- Department of Pharmacy Practice and Administration, Western University of Health Sciences, 309 E 2nd St, Pomona, CA, 91766, USA.
| | - Anandi V Law
- Department of Pharmacy Practice and Administration, Western University of Health Sciences, 309 E 2nd St, Pomona, CA, 91766, USA
| | - Moom R Roosan
- Department of Pharmacy Practice, Chapman University, 9401 Geronimo Rd, Irvine, CA, 92618, USA
| | - Yan Li
- Center for Information Systems and Technology, Claremont Graduate University, 150 E 19th St, Claremont, CA, 91711, USA
| |
Collapse
|
4
|
Kaas-Hansen BS, Placido D, Rodríguez CL, Thorsen-Meyer HC, Gentile S, Nielsen AP, Brunak S, Jürgens G, Andersen SE. Language-agnostic pharmacovigilant text mining to elicit side effects from clinical notes and hospital medication records. Basic Clin Pharmacol Toxicol 2022; 131:282-293. [PMID: 35834334 PMCID: PMC9541191 DOI: 10.1111/bcpt.13773] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Revised: 06/10/2022] [Accepted: 07/09/2022] [Indexed: 11/26/2022]
Abstract
We sought to craft a drug safety signalling pipeline associating latent information in clinical free text with exposures to single drugs and drug pairs. Data arose from 12 secondary and tertiary public hospitals in two Danish regions, comprising approximately half the Danish population. Notes were operationalised with a fastText embedding, based on which we trained 10,720 neural-network models (one for each distinct single-drug/drug-pair exposure) predicting the risk of exposure given an embedding vector. We included 2,905,251 admissions between May 2008 and June 2016, with 13,740,564 distinct drug prescriptions; the median number of prescriptions was 5 (IQR: 3-9) and in 1,184,340 (41%) admissions patients used ≥5 drugs concomitantly. 10,788,259 clinical notes were included, with 179,441,739 tokens retained after pruning. Of 345 single-drug signals reviewed, 28 (8.1%) represented possibly undescribed relationships; 186 (54%) signals were clinically meaningful. 16 (14%) of the 115 drug-pair signals were possible interactions and 2 (1.7%) were known. In conclusion, we built a language-agnostic pipeline for mining associations between free-text information and medication exposure without manual curation, predicting not the likely outcome of a range of exposures, but the likely exposures for outcomes of interest. Our approach may help overcome limitations of text mining methods relying on curated data in English and can help leverage non-English free text for pharmacovigilance.
Collapse
Affiliation(s)
- Benjamin Skov Kaas-Hansen
- Clinical Pharmacology Unit, Zealand University Hospital, Denmark.,NNF Center for Protein Research, University of Copenhagen, Denmark.,Section of Biostatistics, Department of Public Health, University of Copenhagen, Denmark
| | - Davide Placido
- NNF Center for Protein Research, University of Copenhagen, Denmark
| | | | | | | | | | - Søren Brunak
- NNF Center for Protein Research, University of Copenhagen, Denmark
| | - Gesche Jürgens
- Clinical Pharmacology Unit, Zealand University Hospital, Denmark
| | | |
Collapse
|
5
|
Routray R, Tetarenko N, Abu-Assal C, Mockute R, Assuncao B, Chen H, Bao S, Danysz K, Desai S, Cicirello S, Willis V, Alford SH, Krishnamurthy V, Mingle E. Application of Augmented Intelligence for Pharmacovigilance Case Seriousness Determination. Drug Saf 2020; 43:57-66. [PMID: 31605285 PMCID: PMC6965337 DOI: 10.1007/s40264-019-00869-4] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
INTRODUCTION Identification of adverse events and determination of their seriousness ensures timely detection of potential patient safety concerns. Adverse event seriousness is a key factor in defining reporting timelines and is often performed manually by pharmacovigilance experts. The dramatic increase in the volume of safety reports necessitates exploration of scalable solutions that also meet reporting timeline requirements. OBJECTIVE The aim of this study was to develop an augmented intelligence methodology for automatically identifying adverse event seriousness in spontaneous, solicited, and medical literature safety reports. Deep learning models were evaluated for accuracy and/or the F1 score against a ground truth labeled by pharmacovigilance experts. METHODS Using a stratified random sample of safety reports received by Celgene, we developed three neural networks for addressing identification of adverse event seriousness: (1) a binary adverse-event level seriousness classifier; (2) a classifier for determining seriousness categorization at the adverse-event level; and (3) an annotator for identifying seriousness criteria terms to provide supporting evidence at the document level. RESULTS The seriousness classifier achieved an accuracy of 83.0% in post-marketing reports, 92.9% in solicited reports, and 86.3% in medical literature reports. F1 scores for seriousness categorization were 77.7 for death, 78.9 for hospitalization, and 75.5 for important medical events. The seriousness annotator achieved an F1 score of 89.9 in solicited reports, and 75.2 in medical literature reports. CONCLUSIONS The results of this study indicate that a neural network approach can provide an accurate and scalable solution for potentially augmenting pharmacovigilance practitioner determination of adverse event seriousness in spontaneous, solicited, and medical literature reports.
Collapse
|
6
|
Terrier J, Daali Y, Fontana P, Csajka C, Reny JL. Towards Personalized Antithrombotic Treatments: Focus on P2Y 12 Inhibitors and Direct Oral Anticoagulants. Clin Pharmacokinet 2020; 58:1517-1532. [PMID: 31250210 DOI: 10.1007/s40262-019-00792-y] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
Oral anticoagulants and antiplatelet drugs are commonly prescribed to lower the risk of cardiovascular diseases, such as venous and arterial thrombosis, which represent the leading causes of mortality worldwide. A significant percentage of patients taking antithrombotics will nevertheless experience bleeding or recurrent ischemic events, and this represents a major public health issue. Cardiovascular medicine is now questioning the one-size-fits-all policy, and more personalized approaches are increasingly being considered. However, the available tools are currently limited and they are only moderately able to predict clinical events or have a significant impact on clinical outcomes. Predicting concentrations of antithrombotics in blood could be an effective means of personalization as they have been associated with bleeding and recurrent ischemia. Target concentration interventions could take advantage of physiologically based pharmacokinetic (PBPK) and population-based pharmacokinetic (POPPK) models, which are increasingly used in clinical settings and have attracted the interest of governmental regulatory agencies, to propose dosages adapted to specific population characteristics. These models have the benefit of combining parameters from different sources, such as experimental in vitro data and patients' demographic, genetic, and physiological in vivo data, to characterize the dose-concentration relationships of compounds of interest. As such, they can be used to predict individual drug exposure. In the near future, these models could therefore be a valuable means of predicting personalized antithrombotic blood concentrations and, hopefully, of preventing clinical non-response or bleeding in a given patient. Existing approaches for personalization of antithrombotic prescriptions will be reviewed using practical examples for P2Y12 inhibitors and direct oral anticoagulants. The review will additionally focus on the existing PBPK and POPPK models for these two categories of drugs. Lastly, we address potential scenarios for their implementation in clinics, along with the main limitations and challenges.
Collapse
Affiliation(s)
- Jean Terrier
- Division of General Internal Medicine, Geneva University Hospitals, Geneva, Switzerland.,Geneva Platelet Group, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Youssef Daali
- Geneva Platelet Group, Faculty of Medicine, University of Geneva, Geneva, Switzerland.,School of Pharmaceutical Sciences, University of Geneva, University of Lausanne, Geneva, Switzerland.,Clinical Pharmacology and Toxicology Service, Anesthesiology, Pharmacology and Intensive Care Department, Geneva University Hospitals, Geneva, Switzerland
| | - Pierre Fontana
- Geneva Platelet Group, Faculty of Medicine, University of Geneva, Geneva, Switzerland.,Division of Angiology and Haemostasis, Geneva University Hospitals, Geneva, Switzerland
| | - Chantal Csajka
- School of Pharmaceutical Sciences, University of Geneva, University of Lausanne, Geneva, Switzerland
| | - Jean-Luc Reny
- Division of General Internal Medicine, Geneva University Hospitals, Geneva, Switzerland. .,Geneva Platelet Group, Faculty of Medicine, University of Geneva, Geneva, Switzerland. .,Division of Internal Medicine and Rehabilitation, Geneva University Hospitals, Rue Gabrielle-Perret-Gentil 4, 1205, Geneva, Switzerland.
| |
Collapse
|
7
|
da Silva DA, ten Caten CS, dos Santos RP, Fogliatto FS, Hsuan J. Predicting the occurrence of surgical site infections using text mining and machine learning. PLoS One 2019; 14:e0226272. [PMID: 31834905 PMCID: PMC6910696 DOI: 10.1371/journal.pone.0226272] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Accepted: 11/22/2019] [Indexed: 12/11/2022] Open
Abstract
In this study we propose the use of text mining and machine learning methods to predict and detect Surgical Site Infections (SSIs) using textual descriptions of surgeries and post-operative patients’ records, mined from the database of a high complexity University hospital. SSIs are among the most common adverse events experienced by hospitalized patients; preventing such events is fundamental to ensure patients’ safety. Knowledge on SSI occurrence rates may also be useful in preventing future episodes. We analyzed 15,479 surgery descriptions and post-operative records testing different preprocessing strategies and the following machine learning algorithms: Linear SVC, Logistic Regression, Multinomial Naive Bayes, Nearest Centroid, Random Forest, Stochastic Gradient Descent, and Support Vector Classification (SVC). For prediction purposes, the best result was obtained using the Stochastic Gradient Descent method (79.7% ROC-AUC); for detection, Logistic Regression yielded the best performance (80.6% ROC-AUC).
Collapse
Affiliation(s)
- Daniel A. da Silva
- Industrial Engineering Department, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
| | - Carla S. ten Caten
- Industrial Engineering Department, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
| | | | - Flavio S. Fogliatto
- Industrial Engineering Department, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
- * E-mail:
| | | |
Collapse
|
8
|
Liu F, Jagannatha A, Yu H. Towards Drug Safety Surveillance and Pharmacovigilance: Current Progress in Detecting Medication and Adverse Drug Events from Electronic Health Records. Drug Saf 2019; 42:95-97. [PMID: 30649734 DOI: 10.1007/s40264-018-0766-8] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Affiliation(s)
- Feifan Liu
- Department of Quantitative Health Sciences and Radiology, University of Massachusetts Medical School, Worcester, MA, USA
| | - Abhyuday Jagannatha
- College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA
| | - Hong Yu
- College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA. .,Department of Computer Science, University of Massachusetts, 220 Pawtucket St, Lowell, MA, 01854-2874, USA. .,Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA. .,Bedford Veterans Affairs Medical Center, Bedford, MA, USA.
| |
Collapse
|
9
|
Wunnava S, Qin X, Kakar T, Sen C, Rundensteiner EA, Kong X. Adverse Drug Event Detection from Electronic Health Records Using Hierarchical Recurrent Neural Networks with Dual-Level Embedding. Drug Saf 2019; 42:113-122. [PMID: 30649736 DOI: 10.1007/s40264-018-0765-9] [Citation(s) in RCA: 26] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
INTRODUCTION Adverse drug event (ADE) detection is a vital step towards effective pharmacovigilance and prevention of future incidents caused by potentially harmful ADEs. The electronic health records (EHRs) of patients in hospitals contain valuable information regarding ADEs and hence are an important source for detecting ADE signals. However, EHR texts tend to be noisy. Yet applying off-the-shelf tools for EHR text preprocessing jeopardizes the subsequent ADE detection performance, which depends on a well tokenized text input. OBJECTIVE In this paper, we report our experience with the NLP Challenges for Detecting Medication and Adverse Drug Events from Electronic Health Records (MADE1.0), which aims to promote deep innovations on this subject. In particular, we have developed rule-based sentence and word tokenization techniques to deal with the noise in the EHR text. METHODS We propose a detection methodology by adapting a three-layered, deep learning architecture of (1) recurrent neural network [bi-directional long short-term memory (Bi-LSTM)] for character-level word representation to encode the morphological features of the medical terminology, (2) Bi-LSTM for capturing the contextual information of each word within a sentence, and (3) conditional random fields for the final label prediction by also considering the surrounding words. We experiment with different word embedding methods commonly used in word-level classification tasks and demonstrate the impact of an integrated usage of both domain-specific and general-purpose pre-trained word embedding for detecting ADEs from EHRs. RESULTS Our system was ranked first for the named entity recognition task in the MADE1.0 challenge, with a micro-averaged F1-score of 0.8290 (official score). CONCLUSION Our results indicate that the integration of two widely used sequence labeling techniques that complement each other along with dual-level embedding (character level and word level) to represent words in the input layer results in a deep learning architecture that achieves excellent information extraction accuracy for EHR notes.
Collapse
Affiliation(s)
- Susmitha Wunnava
- Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA, 01609, USA.
| | - Xiao Qin
- Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA, 01609, USA
| | - Tabassum Kakar
- Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA, 01609, USA
| | - Cansu Sen
- Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA, 01609, USA
| | | | - Xiangnan Kong
- Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA, 01609, USA
| |
Collapse
|
10
|
Beeksma M, Verberne S, van den Bosch A, Das E, Hendrickx I, Groenewoud S. Predicting life expectancy with a long short-term memory recurrent neural network using electronic medical records. BMC Med Inform Decis Mak 2019; 19:36. [PMID: 30819172 PMCID: PMC6394008 DOI: 10.1186/s12911-019-0775-2] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2018] [Accepted: 02/18/2019] [Indexed: 01/03/2023] Open
Abstract
BACKGROUND Life expectancy is one of the most important factors in end-of-life decision making. Good prognostication for example helps to determine the course of treatment and helps to anticipate the procurement of health care services and facilities, or more broadly: facilitates Advance Care Planning. Advance Care Planning improves the quality of the final phase of life by stimulating doctors to explore the preferences for end-of-life care with their patients, and people close to the patients. Physicians, however, tend to overestimate life expectancy, and miss the window of opportunity to initiate Advance Care Planning. This research tests the potential of using machine learning and natural language processing techniques for predicting life expectancy from electronic medical records. METHODS We approached the task of predicting life expectancy as a supervised machine learning task. We trained and tested a long short-term memory recurrent neural network on the medical records of deceased patients. We developed the model with a ten-fold cross-validation procedure, and evaluated its performance on a held-out set of test data. We compared the performance of a model which does not use text features (baseline model) to the performance of a model which uses features extracted from the free texts of the medical records (keyword model), and to doctors' performance on a similar task as described in scientific literature. RESULTS Both doctors and the baseline model were correct in 20% of the cases, taking a margin of 33% around the actual life expectancy as the target. The keyword model, in comparison, attained an accuracy of 29% with its prognoses. While doctors overestimated life expectancy in 63% of the incorrect prognoses, which harms anticipation to appropriate end-of-life care, the keyword model overestimated life expectancy in only 31% of the incorrect prognoses. CONCLUSIONS Prognostication of life expectancy is difficult for humans. Our research shows that machine learning and natural language processing techniques offer a feasible and promising approach to predicting life expectancy. The research has potential for real-life applications, such as supporting timely recognition of the right moment to start Advance Care Planning.
Collapse
Affiliation(s)
- Merijn Beeksma
- Centre for Language Studies, Radboud University, Erasmusplein 1, 6525 HT Nijmegen, The Netherlands
| | - Suzan Verberne
- Leiden Institute for Advanced Computer Sciences, Leiden University, Niels Bohrweg 1, 2333 CA Leiden, The Netherlands
| | - Antal van den Bosch
- KNAW Meertens Institute, Oudezijds Achterburgwal 185, 1012 DK Amsterdam, The Netherlands
| | - Enny Das
- Centre for Language Studies, Radboud University, Erasmusplein 1, 6525 HT Nijmegen, The Netherlands
| | - Iris Hendrickx
- Centre for Language Studies, Radboud University, Erasmusplein 1, 6525 HT Nijmegen, The Netherlands
| | - Stef Groenewoud
- IQ Healthcare, Radboudumc, Mailbox 9101, 6500 HB Nijmegen, The Netherlands
| |
Collapse
|
11
|
Li F, Liu W, Yu H. Extraction of Information Related to Adverse Drug Events from Electronic Health Record Notes: Design of an End-to-End Model Based on Deep Learning. JMIR Med Inform 2018; 6:e12159. [PMID: 30478023 PMCID: PMC6288593 DOI: 10.2196/12159] [Citation(s) in RCA: 34] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2018] [Revised: 10/31/2018] [Accepted: 11/09/2018] [Indexed: 12/26/2022] Open
Abstract
Background Pharmacovigilance and drug-safety surveillance are crucial for monitoring adverse drug events (ADEs), but the main ADE-reporting systems such as Food and Drug Administration Adverse Event Reporting System face challenges such as underreporting. Therefore, as complementary surveillance, data on ADEs are extracted from electronic health record (EHR) notes via natural language processing (NLP). As NLP develops, many up-to-date machine-learning techniques are introduced in this field, such as deep learning and multi-task learning (MTL). However, only a few studies have focused on employing such techniques to extract ADEs. Objective We aimed to design a deep learning model for extracting ADEs and related information such as medications and indications. Since extraction of ADE-related information includes two steps—named entity recognition and relation extraction—our second objective was to improve the deep learning model using multi-task learning between the two steps. Methods We employed the dataset from the Medication, Indication and Adverse Drug Events (MADE) 1.0 challenge to train and test our models. This dataset consists of 1089 EHR notes of cancer patients and includes 9 entity types such as Medication, Indication, and ADE and 7 types of relations between these entities. To extract information from the dataset, we proposed a deep-learning model that uses a bidirectional long short-term memory (BiLSTM) conditional random field network to recognize entities and a BiLSTM-Attention network to extract relations. To further improve the deep-learning model, we employed three typical MTL methods, namely, hard parameter sharing, parameter regularization, and task relation learning, to build three MTL models, called HardMTL, RegMTL, and LearnMTL, respectively. Results Since extraction of ADE-related information is a two-step task, the result of the second step (ie, relation extraction) was used to compare all models. We used microaveraged precision, recall, and F1 as evaluation metrics. Our deep learning model achieved state-of-the-art results (F1=65.9%), which is significantly higher than that (F1=61.7%) of the best system in the MADE1.0 challenge. HardMTL further improved the F1 by 0.8%, boosting the F1 to 66.7%, whereas RegMTL and LearnMTL failed to boost the performance. Conclusions Deep learning models can significantly improve the performance of ADE-related information extraction. MTL may be effective for named entity recognition and relation extraction, but it depends on the methods, data, and other factors. Our results can facilitate research on ADE detection, NLP, and machine learning.
Collapse
Affiliation(s)
- Fei Li
- Department of Computer Science, University of Massachusetts Lowell, Lowell, MA, United States.,Center for Healthcare Organization and Implementation Research, Bedford Veterans Affairs Medical Center, Bedford, MA, United States.,Department of Medicine, University of Massachusetts Medical School, Worcester, MA, United States
| | - Weisong Liu
- Department of Computer Science, University of Massachusetts Lowell, Lowell, MA, United States.,Center for Healthcare Organization and Implementation Research, Bedford Veterans Affairs Medical Center, Bedford, MA, United States.,Department of Medicine, University of Massachusetts Medical School, Worcester, MA, United States
| | - Hong Yu
- Department of Computer Science, University of Massachusetts Lowell, Lowell, MA, United States.,Center for Healthcare Organization and Implementation Research, Bedford Veterans Affairs Medical Center, Bedford, MA, United States.,Department of Medicine, University of Massachusetts Medical School, Worcester, MA, United States.,School of Computer Science, University of Massachusetts, Amherst, MA, United States
| |
Collapse
|
12
|
Abatemarco D, Perera S, Bao SH, Desai S, Assuncao B, Tetarenko N, Danysz K, Mockute R, Widdowson M, Fornarotto N, Beauchamp S, Cicirello S, Mingle E. Training Augmented Intelligent Capabilities for Pharmacovigilance: Applying Deep-learning Approaches to Individual Case Safety Report Processing. Pharmaceut Med 2018; 32:391-401. [PMID: 30546259 PMCID: PMC6267537 DOI: 10.1007/s40290-018-0251-9] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
Abstract
Introduction Regulations are increasing the scope of activities that fall under the remit of drug safety. Currently, individual case safety report (ICSR) collection and collation is done manually, requiring pharmacovigilance professionals to perform many transactional activities before data are available for assessment and aggregated analyses. For a biopharmaceutical company to meet its responsibilities to patients and regulatory bodies regarding the safe use and distribution of its products, improved business processes must be implemented to drive the industry forward in the best interest of patients globally. Augmented intelligent capabilities have already demonstrated success in capturing adverse events from diverse data sources. It has potential to provide a scalable solution for handling the ever-increasing ICSR volumes experienced within the industry by supporting pharmacovigilance professionals’ decision-making. Objective The aim of this study was to train and evaluate a consortium of cognitive services to identify key characteristics of spontaneous ICSRs satisfying an acceptable level of accuracy determined by considering business requirements and effective use in a real-world setting. The results of this study will serve as supporting evidence for or against implementing augmented intelligence in case processing to increase operational efficiency and data quality consistency. Methods A consortium of ten cognitive services to augment aspects of ICSR processing were identified and trained through deep-learning approaches. The input data for model training were 20,000 ICSRs received by Celgene drug safety over a 2-year period. The data were manually made machine-readable through the process of transcription, which converts images into text. The machine-readable documents were manually annotated for pharmacovigilance data elements to facilitate the training and testing of the cognitive services. Once trained by cognitive developers, the cognitive services’ output was reviewed by pharmacovigilance subject-matter experts against the accepted ground-truth for correctness and completeness. To be considered adequately trained and functional, each cognitive service was required to reach a threshold of F1 or accuracy score ≥ 75%. Results All ten cognitive services under development have reached an evaluative score ≥ 75% for spontaneous ICSRs. Conclusion All cognitive services under development have achieved the minimum evaluative threshold to be considered adequately trained, demonstrating how machine-learning and natural language processing techniques together provide accurate outputs that may augment pharmacovigilance professionals’ processing of spontaneous ICSRs quickly and accurately. The intention of augmented intelligence is not to replace the pharmacovigilance professional, but rather support them in their consistent decision-making so that they may better handle the overwhelming amount of data otherwise manually curated and monitored for ongoing drug surveillance requirements. Through this supported decision-making, pharmacovigilance professionals may have more time to apply their knowledge in assessing the case rather than spending it performing transactional tasks to simply capture the pertinent data within a safety database. By capturing data consistently and efficiently, we begin to build a corpus of data upon which analyses may be conducted and insights gleaned. Cognitive services may be key to an organization’s transformation to more proactive decision-making needed to meet regulatory requirements and enhance patient safety.
Collapse
Affiliation(s)
| | - Sujan Perera
- IBM Watson Health, 75 Binney Street, Cambridge, MA 02142 USA
| | - Sheng Hua Bao
- IBM Watson Health, 75 Binney Street, Cambridge, MA 02142 USA
| | - Sameen Desai
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| | - Bruno Assuncao
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| | - Niki Tetarenko
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| | - Karolina Danysz
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| | - Ruta Mockute
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| | - Mark Widdowson
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| | | | | | | | - Edward Mingle
- 1Celgene Corporation, 86 Morris Avenue, Summit, NJ 07901 USA
| |
Collapse
|
13
|
Munkhdalai T, Liu F, Yu H. Clinical Relation Extraction Toward Drug Safety Surveillance Using Electronic Health Record Narratives: Classical Learning Versus Deep Learning. JMIR Public Health Surveill 2018; 4:e29. [PMID: 29695376 PMCID: PMC5943628 DOI: 10.2196/publichealth.9361] [Citation(s) in RCA: 37] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2017] [Revised: 02/03/2018] [Accepted: 02/05/2018] [Indexed: 12/20/2022] Open
Abstract
BACKGROUND Medication and adverse drug event (ADE) information extracted from electronic health record (EHR) notes can be a rich resource for drug safety surveillance. Existing observational studies have mainly relied on structured EHR data to obtain ADE information; however, ADEs are often buried in the EHR narratives and not recorded in structured data. OBJECTIVE To unlock ADE-related information from EHR narratives, there is a need to extract relevant entities and identify relations among them. In this study, we focus on relation identification. This study aimed to evaluate natural language processing and machine learning approaches using the expert-annotated medical entities and relations in the context of drug safety surveillance, and investigate how different learning approaches perform under different configurations. METHODS We have manually annotated 791 EHR notes with 9 named entities (eg, medication, indication, severity, and ADEs) and 7 different types of relations (eg, medication-dosage, medication-ADE, and severity-ADE). Then, we explored 3 supervised machine learning systems for relation identification: (1) a support vector machines (SVM) system, (2) an end-to-end deep neural network system, and (3) a supervised descriptive rule induction baseline system. For the neural network system, we exploited the state-of-the-art recurrent neural network (RNN) and attention models. We report the performance by macro-averaged precision, recall, and F1-score across the relation types. RESULTS Our results show that the SVM model achieved the best average F1-score of 89.1% on test data, outperforming the long short-term memory (LSTM) model with attention (F1-score of 65.72%) as well as the rule induction baseline system (F1-score of 7.47%) by a large margin. The bidirectional LSTM model with attention achieved the best performance among different RNN models. With the inclusion of additional features in the LSTM model, its performance can be boosted to an average F1-score of 77.35%. CONCLUSIONS It shows that classical learning models (SVM) remains advantageous over deep learning models (RNN variants) for clinical relation identification, especially for long-distance intersentential relations. However, RNNs demonstrate a great potential of significant improvement if more training data become available. Our work is an important step toward mining EHRs to improve the efficacy of drug safety surveillance. Most importantly, the annotated data used in this study will be made publicly available, which will further promote drug safety research in the community.
Collapse
Affiliation(s)
- Tsendsuren Munkhdalai
- Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA, United States
| | - Feifan Liu
- Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA, United States
| | - Hong Yu
- Department of Computer Science, University of Massachusetts Lowell, Lowell, MA, United States.,The Bedford Veterans Affairs Medical Center, Bedford, MA, United States
| |
Collapse
|
14
|
Alvaro N, Miyao Y, Collier N. TwiMed: Twitter and PubMed Comparable Corpus of Drugs, Diseases, Symptoms, and Their Relations. JMIR Public Health Surveill 2017; 3:e24. [PMID: 28468748 PMCID: PMC5438461 DOI: 10.2196/publichealth.6396] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2016] [Revised: 11/24/2016] [Accepted: 03/20/2017] [Indexed: 12/11/2022] Open
Abstract
BACKGROUND Work on pharmacovigilance systems using texts from PubMed and Twitter typically target at different elements and use different annotation guidelines resulting in a scenario where there is no comparable set of documents from both Twitter and PubMed annotated in the same manner. OBJECTIVE This study aimed to provide a comparable corpus of texts from PubMed and Twitter that can be used to study drug reports from these two sources of information, allowing researchers in the area of pharmacovigilance using natural language processing (NLP) to perform experiments to better understand the similarities and differences between drug reports in Twitter and PubMed. METHODS We produced a corpus comprising 1000 tweets and 1000 PubMed sentences selected using the same strategy and annotated at entity level by the same experts (pharmacists) using the same set of guidelines. RESULTS The resulting corpus, annotated by two pharmacists, comprises semantically correct annotations for a set of drugs, diseases, and symptoms. This corpus contains the annotations for 3144 entities, 2749 relations, and 5003 attributes. CONCLUSIONS We present a corpus that is unique in its characteristics as this is the first corpus for pharmacovigilance curated from Twitter messages and PubMed sentences using the same data selection and annotation strategies. We believe this corpus will be of particular interest for researchers willing to compare results from pharmacovigilance systems (eg, classifiers and named entity recognition systems) when using data from Twitter and from PubMed. We hope that given the comprehensive set of drug names and the annotated entities and relations, this corpus becomes a standard resource to compare results from different pharmacovigilance studies in the area of NLP.
Collapse
Affiliation(s)
- Nestor Alvaro
- National Institute of Informatics, Department of Informatics, Tokyo, Japan
- The Graduate University for Advanced Studies (SOKENDAI), Kanagawa, Japan
| | - Yusuke Miyao
- National Institute of Informatics, Department of Informatics, Tokyo, Japan
- The Graduate University for Advanced Studies (SOKENDAI), Kanagawa, Japan
| | - Nigel Collier
- Faculty of Modern & Medieval Languages, Department of Theoretical and Applied Linguistics, University of Cambridge, Cambridge, United Kingdom
| |
Collapse
|
15
|
Bidirectional RNN for Medical Event Detection in Electronic Health Records. PROCEEDINGS OF THE CONFERENCE. ASSOCIATION FOR COMPUTATIONAL LINGUISTICS. NORTH AMERICAN CHAPTER. MEETING 2016; 2016:473-482. [PMID: 27885364 DOI: 10.18653/v1/n16-1056] [Citation(s) in RCA: 99] [Impact Index Per Article: 12.4] [Reference Citation Analysis] [Abstract] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Sequence labeling for extraction of medical events and their attributes from unstructured text in Electronic Health Record (EHR) notes is a key step towards semantic understanding of EHRs. It has important applications in health informatics including pharmacovigilance and drug surveillance. The state of the art supervised machine learning models in this domain are based on Conditional Random Fields (CRFs) with features calculated from fixed context windows. In this application, we explored recurrent neural network frameworks and show that they significantly out-performed the CRF models.
Collapse
|
16
|
Nikfarjam A, Sarker A, O'Connor K, Ginn R, Gonzalez G. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. J Am Med Inform Assoc 2015; 22:671-81. [PMID: 25755127 PMCID: PMC4457113 DOI: 10.1093/jamia/ocu041] [Citation(s) in RCA: 221] [Impact Index Per Article: 24.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2014] [Accepted: 12/04/2014] [Indexed: 02/06/2023] Open
Abstract
Objective Social media is becoming increasingly popular as a platform for sharing personal health-related information. This information can be utilized for public health monitoring tasks, particularly for pharmacovigilance, via the use of natural language processing (NLP) techniques. However, the language in social media is highly informal, and user-expressed medical concepts are often nontechnical, descriptive, and challenging to extract. There has been limited progress in addressing these challenges, and thus far, advanced machine learning-based NLP techniques have been underutilized. Our objective is to design a machine learning-based approach to extract mentions of adverse drug reactions (ADRs) from highly informal text in social media. Methods We introduce ADRMine, a machine learning-based concept extraction system that uses conditional random fields (CRFs). ADRMine utilizes a variety of features, including a novel feature for modeling words’ semantic similarities. The similarities are modeled by clustering words based on unsupervised, pretrained word representation vectors (embeddings) generated from unlabeled user posts in social media using a deep learning technique. Results ADRMine outperforms several strong baseline systems in the ADR extraction task by achieving an F-measure of 0.82. Feature analysis demonstrates that the proposed word cluster features significantly improve extraction performance. Conclusion It is possible to extract complex medical concepts, with relatively high performance, from informal, user-generated content. Our approach is particularly scalable, suitable for social media mining, as it relies on large volumes of unlabeled data, thus diminishing the need for large, annotated training data sets.
Collapse
Affiliation(s)
- Azadeh Nikfarjam
- Department of Biomedical Informatics, Arizona State University, Scottsdale, AZ, USA
| | - Abeed Sarker
- Department of Biomedical Informatics, Arizona State University, Scottsdale, AZ, USA
| | - Karen O'Connor
- Department of Biomedical Informatics, Arizona State University, Scottsdale, AZ, USA
| | - Rachel Ginn
- Department of Biomedical Informatics, Arizona State University, Scottsdale, AZ, USA
| | - Graciela Gonzalez
- Department of Biomedical Informatics, Arizona State University, Scottsdale, AZ, USA
| |
Collapse
|