101
|
Diaz-Garelli F, Lenoir KM, Wells BJ. Catch Me if You Can: Acute Events Hidden in Structured Chronic Disease Diagnosis Descriptions Show Detectable Recording Patterns in EHR. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2021; 2020:373-382. [PMID: 33936410 PMCID: PMC8075503] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Our previous research shows that structured cancer DX description data accuracy varied across electronic health record (EHR) segments (e.g. encounter DX, problem list, etc.). We provide initial evidence corroborating these findings in EHRs from patients with diabetes. We hypothesized that the odds of recording an "uncontrolled diabetes" DX increased after a hemoglobin A1c result above 9% and that this rate would vary across EHR segments. Our statistical models revealed that each DX indicating uncontrolled diabetes was 2.6% more likely to occur post-A1c>9% overall (adj-p=.0005) and 3.9% after controlling for EHR segment (adj-p<.0001). However, odds ratios varied across segments (1.021
Collapse
|
102
|
Soni S, Roberts K. Patient Cohort Retrieval using Transformer Language Models. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2021; 2020:1150-1159. [PMID: 33936491 PMCID: PMC8075458] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
We apply deep learning-based language models to the task of patient cohort retrieval (CR) with the aim to assess their efficacy. The task ofCR requires the extraction of relevant documents from the electronic health records (EHRs) on the basis of a given query. Given the recent advancements in the field of document retrieval, we map the task of CR to a document retrieval task and apply various deep neural models implemented for the general domain tasks. In this paper, we propose a framework for retrieving patient cohorts using neural language models without the need of explicit feature engineering and domain expertise. We find that a majority of our models outperform the BM25 baseline method on various evaluation metrics.
Collapse
Affiliation(s)
- Sarvesh Soni
- School of Biomedical Informatics The University of Texas Health Science Center at Houston Houston TX, USA
| | - Kirk Roberts
- School of Biomedical Informatics The University of Texas Health Science Center at Houston Houston TX, USA
| |
Collapse
|
103
|
Rosen T, Platts-Mills TF, Fulmer T. Screening for elder mistreatment in emergency departments: current progress and recommendations for next steps. J Elder Abuse Negl 2021; 32:295-315. [PMID: 32508284 DOI: 10.1080/08946566.2020.1768997] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
Emergency Department (ED) visits provide an important but seldom realized opportunity to identify elder mistreatment. Many screening tools exist, including several that are brief and may be effective, but few have been specifically designed for or tested in EDs. In addition to the absence of validated tools, other challenges with implementing ED elder mistreatment screening include difficulty integrating anything longer than a few questions into a busy clinical encounter and resources required to respond to positive screens. The Electronic Health Record (EHR) offers a critical tool to facilitate elder mistreatment screening through required data entry and real-time monitoring of compliance and results. We describe current work in the field and recommend next steps including design and testing of a two-step screening process, implementation research to accelerate adoption, development of ED-based interventions and referral protocols for positive cases, and consideration of the important role of pre-hospital providers in case identification.
Collapse
Affiliation(s)
- Tony Rosen
- Department of Emergency Medicine, Weill Cornell Medicine / NewYork-Presbyterian Hospital , New York, NY, USA
| | - Timothy F Platts-Mills
- Department of Emergency Medicine, University of North Carolina School of Medicine , Chapel Hill, North Carolina, USA
| | - Terry Fulmer
- The John A. Hartford Foundation , New York, NY, USA
| |
Collapse
|
104
|
Rashidian S, Abell-Hart K, Hajagos J, Moffitt R, Lingam V, Garcia V, Tsai CW, Wang F, Dong X, Sun S, Deng J, Gupta R, Miller J, Saltz J, Saltz M. Detecting Miscoded Diabetes Diagnosis Codes in Electronic Health Records for Quality Improvement: Temporal Deep Learning Approach. JMIR Med Inform 2020; 8:e22649. [PMID: 33331828 PMCID: PMC7775195 DOI: 10.2196/22649] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2020] [Revised: 09/24/2020] [Accepted: 09/27/2020] [Indexed: 01/16/2023] Open
Abstract
BACKGROUND Diabetes affects more than 30 million patients across the United States. With such a large disease burden, even a small error in classification can be significant. Currently billing codes, assigned at the time of a medical encounter, are the "gold standard" reflecting the actual diseases present in an individual, and thus in aggregate reflect disease prevalence in the population. These codes are generated by highly trained coders and by health care providers but are not always accurate. OBJECTIVE This work provides a scalable deep learning methodology to more accurately classify individuals with diabetes across multiple health care systems. METHODS We leveraged a long short-term memory-dense neural network (LSTM-DNN) model to identify patients with or without diabetes using data from 5 acute care facilities with 187,187 patients and 275,407 encounters, incorporating data elements including laboratory test results, diagnostic/procedure codes, medications, demographic data, and admission information. Furthermore, a blinded physician panel reviewed discordant cases, providing an estimate of the total impact on the population. RESULTS When predicting the documented diagnosis of diabetes, our model achieved an 84% F1 score, 96% area under the curve-receiver operating characteristic curve, and 91% average precision on a heterogeneous data set from 5 distinct health facilities. However, in 81% of cases where the model disagreed with the documented phenotype, a blinded physician panel agreed with the model. Taken together, this suggests that 4.3% of our studied population have either missing or improper diabetes diagnosis. CONCLUSIONS This study demonstrates that deep learning methods can improve clinical phenotyping even when patient data are noisy, sparse, and heterogeneous.
Collapse
Affiliation(s)
- Sina Rashidian
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
| | - Kayley Abell-Hart
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Janos Hajagos
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Richard Moffitt
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Veena Lingam
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Victor Garcia
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Chao-Wei Tsai
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Fusheng Wang
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Xinyu Dong
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
| | - Siao Sun
- Department of Applied Mathematics and Statistics, Stony Brook University, Stony Brook, NY, United States
| | - Jianyuan Deng
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Joshua Miller
- Department of Medicine, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Joel Saltz
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| | - Mary Saltz
- Department of Biomedical Informatics, Renaissance School of Medicine at Stony Brook, Stony Brook, NY, United States
| |
Collapse
|
105
|
Thangaraj PM, Kummer BR, Lorberbaum T, Elkind MSV, Tatonetti NP. Comparative analysis, applications, and interpretation of electronic health record-based stroke phenotyping methods. BioData Min 2020; 13:21. [PMID: 33372632 PMCID: PMC7720570 DOI: 10.1186/s13040-020-00230-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2020] [Accepted: 11/15/2020] [Indexed: 01/14/2023] Open
Abstract
BACKGROUND Accurate identification of acute ischemic stroke (AIS) patient cohorts is essential for a wide range of clinical investigations. Automated phenotyping methods that leverage electronic health records (EHRs) represent a fundamentally new approach cohort identification without current laborious and ungeneralizable generation of phenotyping algorithms. We systematically compared and evaluated the ability of machine learning algorithms and case-control combinations to phenotype acute ischemic stroke patients using data from an EHR. MATERIALS AND METHODS Using structured patient data from the EHR at a tertiary-care hospital system, we built and evaluated machine learning models to identify patients with AIS based on 75 different case-control and classifier combinations. We then estimated the prevalence of AIS patients across the EHR. Finally, we externally validated the ability of the models to detect AIS patients without AIS diagnosis codes using the UK Biobank. RESULTS Across all models, we found that the mean AUROC for detecting AIS was 0.963 ± 0.0520 and average precision score 0.790 ± 0.196 with minimal feature processing. Classifiers trained with cases with AIS diagnosis codes and controls with no cerebrovascular disease codes had the best average F1 score (0.832 ± 0.0383). In the external validation, we found that the top probabilities from a model-predicted AIS cohort were significantly enriched for AIS patients without AIS diagnosis codes (60-150 fold over expected). CONCLUSIONS Our findings support machine learning algorithms as a generalizable way to accurately identify AIS patients without using process-intensive manual feature curation. When a set of AIS patients is unavailable, diagnosis codes may be used to train classifier models.
Collapse
Affiliation(s)
- Phyllis M Thangaraj
- Department of Biomedical Informatics, Columbia University, 622 W 168th St., PH-20, New York, NY, 10032, USA
- Department of Systems Biology, Columbia University, New York, NY, USA
| | - Benjamin R Kummer
- Department of Neurology, Icahn School of Medicine at Mt. Sinai, New York, NY, USA
| | - Tal Lorberbaum
- Department of Biomedical Informatics, Columbia University, 622 W 168th St., PH-20, New York, NY, 10032, USA
- Department of Systems Biology, Columbia University, New York, NY, USA
| | - Mitchell S V Elkind
- Department of Neurology, Vagelos College of Physicians and Surgeons, Columbia University, New York, NY, USA
- Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, NY, USA
| | - Nicholas P Tatonetti
- Department of Biomedical Informatics, Columbia University, 622 W 168th St., PH-20, New York, NY, 10032, USA.
- Department of Systems Biology, Columbia University, New York, NY, USA.
| |
Collapse
|
106
|
Huang S, Huang J, Cai T, Dahal KP, Cagan A, He Z, Stratton J, Gorelik I, Hong C, Cai T, Liao KP. Impact of ICD10 and secular changes on electronic medical record rheumatoid arthritis algorithms. Rheumatology (Oxford) 2020; 59:3759-3766. [PMID: 32413107 DOI: 10.1093/rheumatology/keaa198] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2019] [Accepted: 03/17/2020] [Indexed: 12/18/2022] Open
Abstract
OBJECTIVE The objective of this study was to compare the performance of an RA algorithm developed and trained in 2010 utilizing natural language processing and machine learning, using updated data containing ICD10, new RA treatments, and a new electronic medical records (EMR) system. METHODS We extracted data from subjects with ≥1 RA International Classification of Diseases (ICD) codes from the EMR of two large academic centres to create a data mart. Gold standard RA cases were identified from reviewing a random 200 subjects from the data mart, and a random 100 subjects who only have RA ICD10 codes. We compared the performance of the following algorithms using the original 2010 data with updated data: (i) a published 2010 RA algorithm; (ii) updated algorithm, incorporating ICD10 RA codes and new DMARDs; and (iii) published algorithm using ICD codes only, ICD RA code ≥3. RESULTS The gold standard RA cases had mean age 65.5 years, 78.7% female, 74.1% RF or antibodies to cyclic citrullinated peptide (anti-CCP) positive. The positive predictive value (PPV) for ≥3 RA ICD was 54%, compared with 56% in 2010. At a specificity of 95%, the PPV of the 2010 algorithm and the updated version were both 91%, compared with 94% (95% CI: 91, 96%) in 2010. In subjects with ICD10 data only, the PPV for the updated 2010 RA algorithm was 93%. CONCLUSION The 2010 RA algorithm validated with the updated data with similar performance characteristics as the 2010 data. While the 2010 algorithm continued to perform better than the rule-based approach, the PPV of the latter also remained stable over time.
Collapse
Affiliation(s)
- Sicong Huang
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital.,Department of Medicine, Harvard Medical School
| | - Jie Huang
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital
| | - Tianrun Cai
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital.,Department of Medicine, Harvard Medical School
| | - Kumar P Dahal
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital
| | - Andrew Cagan
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital.,Research Information Science and Computing, Partners Healthcare
| | - Zeling He
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital
| | - Jacklyn Stratton
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital
| | - Isaac Gorelik
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital
| | - Chuan Hong
- Department of Biomedical Informatics, Harvard Medical School.,Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Tianxi Cai
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital.,Department of Biomedical Informatics, Harvard Medical School.,Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Katherine P Liao
- Division of Rheumatology, Inflammation, and Immunity, Brigham and Women's Hospital.,Department of Biomedical Informatics, Harvard Medical School
| |
Collapse
|
107
|
Dligach D, Afshar M, Miller T. Pre-training phenotyping classifiers. J Biomed Inform 2020; 113:103626. [PMID: 33259943 DOI: 10.1016/j.jbi.2020.103626] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2020] [Revised: 09/09/2020] [Accepted: 11/14/2020] [Indexed: 11/17/2022]
Abstract
Recent transformer-based pre-trained language models have become a de facto standard for many text classification tasks. Nevertheless, their utility in the clinical domain, where classification is often performed at encounter or patient level, is still uncertain due to the limitation on the maximum length of input. In this work, we introduce a self-supervised method for pre-training that relies on a masked token objective and is free from the limitation on the maximum input length. We compare the proposed method with supervised pre-training that uses billing codes as a source of supervision. We evaluate the proposed method on one publicly-available and three in-house datasets using the standard evaluation metrics such as the area under the ROC curve and F1 score. We find that, surprisingly, even though self-supervised pre-training performs slightly worse than supervised, it still preserves most of the gains from pre-training.
Collapse
Affiliation(s)
- Dmitriy Dligach
- Loyola University Chicago, Department of Computer Science, Chicago, IL, United States.
| | - Majid Afshar
- Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison, Madison, WI, United States.
| | - Timothy Miller
- Computational Health Informatics Program (CHIP), Boston Children's Hospital and Harvard Medical School, Boston, MA, United States.
| |
Collapse
|
108
|
Capurro D, Barbe M, Daza C, Santa Maria J, Trincado J. Temporal Design Patterns for Digital Phenotype Cohort Selection in Critical Care: Systematic Literature Assessment and Qualitative Synthesis. JMIR Med Inform 2020; 8:e6924. [PMID: 33231554 PMCID: PMC7723741 DOI: 10.2196/medinform.6924] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Revised: 08/30/2020] [Accepted: 10/28/2020] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Inclusion criteria for observational studies frequently contain temporal entities and relations. The use of digital phenotypes to create cohorts in electronic health record-based observational studies requires rich functionality to capture these temporal entities and relations. However, such functionality is not usually available or requires complex database queries and specialized expertise to build them. OBJECTIVE The purpose of this study is to systematically assess observational studies reported in critical care literature to capture design requirements and functionalities for a graphical temporal abstraction-based digital phenotyping tool. METHODS We iteratively extracted attributes describing patients, interventions, and clinical outcomes. We qualitatively synthesized studies, identifying all temporal and nontemporal entities and relations. RESULTS We extracted data from 28 primary studies and 367 temporal and nontemporal entities. We generated a synthesis of entities, relations, and design patterns. CONCLUSIONS We report on the observed types of clinical temporal entities and their relations as well as design requirements for a temporal abstraction-based digital phenotyping system. The results can be used to inform the development of such a system.
Collapse
Affiliation(s)
- Daniel Capurro
- School of Computing and Information Systems, Centre for Digital Transformation of Health, University of Melbourne, Melbourne, Australia.,Department of Internal Medicine, School of Medicine, Pontificia Universidad Catolica de Chile, Santiago, Chile
| | - Mario Barbe
- Department of Biomedical Informatics, Clinica Alemana, Santiago, Chile.,Instituto de Ciencias e Innovación en Medicina, Facultad de Medicina Clínica Alemana, Universidad del Desarrollo, Santiago, Chile
| | - Claudio Daza
- Department of Internal Medicine, School of Medicine, Pontificia Universidad Catolica de Chile, Santiago, Chile
| | - Josefa Santa Maria
- Department of Internal Medicine, School of Medicine, Pontificia Universidad Catolica de Chile, Santiago, Chile
| | - Javier Trincado
- Department of Internal Medicine, School of Medicine, Pontificia Universidad Catolica de Chile, Santiago, Chile
| |
Collapse
|
109
|
Development and evaluation of MADDIE: Method to Acquire Delivery Date Information from Electronic health records. Int J Med Inform 2020; 145:104339. [PMID: 33232918 DOI: 10.1016/j.ijmedinf.2020.104339] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2020] [Revised: 10/29/2020] [Accepted: 11/03/2020] [Indexed: 12/11/2022]
Abstract
OBJECTIVE To develop an algorithm that infers patient delivery dates (PDDs) and delivery-specific details from Electronic Health Records (EHRs) with high accuracy; enabling pregnancy-level outcome studies in women's health. MATERIALS AND METHODS We obtained EHR data from 1,060,100 female patients treated at Penn Medicine hospitals or outpatient clinics between 2010-2017. We developed an algorithm called MADDIE: Method to Acquire Delivery Date Information from Electronic Health Records that infers a PDD for distinct deliveries based on EHR encounter dates assigned a delivery code, the frequency of code usage, and the time differential between code assignments. We validated MADDIE's PDDs against a birth log independently maintained by the Department of Obstetrics and Gynecology. RESULTS MADDIE identified 50,560 patients having 63,334 distinct deliveries. MADDIE was 98.6 % accurate (F1-score 92.1 %) when compared to the birth log. The PDD was on average 0.68 days earlier than the true delivery date for patients with only one delivery (± 1.43 days) and 0.52 days earlier for patients with more than one delivery episode (± 1.11 days). DISCUSSION MADDIE is the first algorithm to successfully infer PDD information using only structured delivery codes and identify multiple deliveries per patient. MADDIE is also the first to validate the accuracy of the PDD using an external gold standard of known delivery dates as opposed to manual chart review of a sample. CONCLUSION MADDIE augments the EHR with delivery-specific details extracted with high accuracy and relies only on structured EHR elements while harnessing temporal information and the frequency of code usage to identify accurate PDDs.
Collapse
|
110
|
Wagholikar KB, Estiri H, Murphy M, Murphy SN. Polar labeling: silver standard algorithm for training disease classifiers. Bioinformatics 2020; 36:3200-3206. [PMID: 32049335 PMCID: PMC7214041 DOI: 10.1093/bioinformatics/btaa088] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2019] [Revised: 01/30/2020] [Accepted: 02/04/2020] [Indexed: 01/29/2023] Open
Abstract
MOTIVATION Expert-labeled data are essential to train phenotyping algorithms for cohort identification. However expert labeling is time and labor intensive, and the costs remain prohibitive for scaling phenotyping to wider use-cases. RESULTS We present an approach referred to as polar labeling (PL), to create silver standard for training machine learning (ML) for disease classification. We test the hypothesis that ML models trained on the silver standard created by applying PL on unlabeled patient records, are comparable in performance to the ML models trained on gold standard, created by clinical experts through manual review of patient records. We perform experimental validation using health records of 38 023 patients spanning six diseases. Our results demonstrate the superior performance of the proposed approach. AVAILABILITY AND IMPLEMENTATION We provide a Python implementation of the algorithm and the Python code developed for this study on Github. SUPPLEMENTARY INFORMATION Supplementary data are available at Bioinformatics online.
Collapse
Affiliation(s)
| | - Hossein Estiri
- Laboratory of Computer Science, Massachusetts General Hospital, Boston, MA 02114, USA
| | | | - Shawn N Murphy
- Laboratory of Computer Science, Massachusetts General Hospital, Boston, MA 02114, USA
| |
Collapse
|
111
|
Ong M, Klann JG, Lin KJ, Maron BA, Murphy SN, Natter MD, Mandl KD. Claims-Based Algorithms for Identifying Patients With Pulmonary Hypertension: A Comparison of Decision Rules and Machine-Learning Approaches. J Am Heart Assoc 2020; 9:e016648. [PMID: 32990147 PMCID: PMC7792386 DOI: 10.1161/jaha.120.016648] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
Background Real‐world healthcare data are an important resource for epidemiologic research. However, accurate identification of patient cohorts—a crucial first step underpinning the validity of research results—remains a challenge. We developed and evaluated claims‐based case ascertainment algorithms for pulmonary hypertension (PH), comparing conventional decision rules with state‐of‐the‐art machine‐learning approaches. Methods and Results We analyzed an electronic health record‐Medicare linked database from two large academic tertiary care hospitals (years 2007–2013). Electronic health record charts were reviewed to form a gold standard cohort of patients with (n=386) and without PH (n=164). Using health encounter data captured in Medicare claims (including patients’ demographics, diagnoses, medications, and procedures), we developed and compared 2 approaches for identifying patients with PH: decision rules and machine‐learning algorithms using penalized lasso regression, random forest, and gradient boosting machine. The most optimal rule‐based algorithm—having ≥3 PH‐related healthcare encounters and having undergone right heart catheterization—attained an area under the receiver operating characteristic curve of 0.64 (sensitivity, 0.75; specificity, 0.48). All 3 machine‐learning algorithms outperformed the most optimal rule‐based algorithm (P<0.001). A model derived from the random forest algorithm achieved an area under the receiver operating characteristic curve of 0.88 (sensitivity, 0.87; specificity, 0.70), and gradient boosting machine achieved comparable results (area under the receiver operating characteristic curve, 0.85; sensitivity, 0.87; specificity, 0.70). Penalized lasso regression achieved an area under the receiver operating characteristic curve of 0.73 (sensitivity, 0.70; specificity, 0.68). Conclusions Research‐grade case identification algorithms for PH can be derived and rigorously validated using machine‐learning algorithms. Simple decision rules commonly applied in published literature performed poorly; more complex rule‐based algorithms may potentially address the limitation of this approach. PH research using claims data would be considerably strengthened through the use of validated algorithms for cohort ascertainment.
Collapse
Affiliation(s)
- Mei‐Sing Ong
- Department of Population MedicineHarvard Medical School &Harvard Pilgrim Health Care InstituteBostonMA
- Computational Health Informatics ProgramBoston Children’s HospitalBostonMA
| | - Jeffrey G. Klann
- Laboratory of Computer ScienceMassachusetts General HospitalHarvard Medical SchoolBostonMA
| | - Kueiyu Joshua Lin
- Division of Pharmacoepidemiology and PharmacoeconomicsDepartment of MedicineBrigham and Women’s HospitalHarvard Medical SchoolBostonMA
| | - Bradley A. Maron
- Cardiovascular DivisionDepartment of MedicineBrigham and Women’s HospitalHarvard Medical SchoolBostonMA
| | - Shawn N. Murphy
- Department of NeurologyMassachusetts General Hospital, Harvard Medical SchoolBostonMA
| | - Marc D. Natter
- Computational Health Informatics ProgramBoston Children’s HospitalBostonMA
- Department of PediatricsHarvard Medical SchoolBostonMA
| | - Kenneth D. Mandl
- Computational Health Informatics ProgramBoston Children’s HospitalBostonMA
- Department of PediatricsHarvard Medical SchoolBostonMA
- Department of Biomedical InformaticsHarvard Medical SchoolBostonMA
| |
Collapse
|
112
|
Sangal RB, Fodeh S, Taylor A, Rothenberg C, Finn EB, Sheth K, Matouk C, Ulrich A, Parwani V, Sather J, Venkatesh A. Identification of Patients with Nontraumatic Intracranial Hemorrhage Using Administrative Claims Data. J Stroke Cerebrovasc Dis 2020; 29:105306. [PMID: 33070110 DOI: 10.1016/j.jstrokecerebrovasdis.2020.105306] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2020] [Revised: 09/02/2020] [Accepted: 09/05/2020] [Indexed: 10/23/2022] Open
Abstract
INTRODUCTION Nontraumatic intracranial hemorrhage (ICH) is a neurological emergency of research interest; however, unlike ischemic stroke, has not been well studied in large datasets due to the lack of an established administrative claims-based definition. We aimed to evaluate both explicit diagnosis codes and machine learning methods to create a claims-based definition for this clinical phenotype. METHODS We examined all patients admitted to our tertiary medical center with a primary or secondary International Classification of Disease version 9 (ICD-9) or 10 (ICD-10) code for ICH in claims from any portion of the hospitalization in 2014-2015. As a gold standard, we defined the nontraumatic ICH phenotype based on manual chart review. We tested explicit definitions based on ICD-9 and ICD-10 that had been previously published in the literature as well as four machine learning classifiers including support vector machine (SVM), logistic regression with LASSO, random forest and xgboost. We report five standard measures of model performance for each approach. RESULTS A total of 1830 patients with 2145 unique ICD-10 codes were included in the initial dataset, of which 437 (24%) were true positive based on manual review. The explicit ICD-10 definition performed best (Sensitivity = 0.89 (95% CI 0.85-0.92), Specificity = 0.83 (0.81-0.85), F-score = 0.73 (0.69-0.77)) and improves on an explicit ICD-9 definition (Sensitivity = 0.87 (0.83-0.90), Specificity = 0.77 (0.74-0.79), F-score = 0.67 (0.63-0.71). Among machine learning classifiers, SVM performed best (Sensitivity = 0.78 (0.75-0.82), Specificity = 0.84 (0.81-0.87), AUC = 0.89 (0.87-0.92), F-score = 0.66 (0.62-0.69)). CONCLUSIONS An explicit ICD-10 definition can be used to accurately identify patients with a nontraumatic ICH phenotype with substantially better performance than ICD-9. An explicit ICD-10 based definition is easier to implement and quantitatively not appreciably improved with the additional application of machine learning classifiers. Future research utilizing large datasets should utilize this definition to address important research gaps.
Collapse
Affiliation(s)
- Rohit B Sangal
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Samah Fodeh
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Andrew Taylor
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Craig Rothenberg
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Emily B Finn
- Department of Internal Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Kevin Sheth
- Department of Neurology, United States; Yale University School of Medicine, New Haven CT, United States
| | | | - Andrew Ulrich
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Vivek Parwani
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - John Sather
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States
| | - Arjun Venkatesh
- Department of Emergency Medicine, United States; Yale University School of Medicine, New Haven CT, United States.
| |
Collapse
|
113
|
Obeid JS, Davis M, Turner M, Meystre SM, Heider PM, O'Bryan EC, Lenert LA. An artificial intelligence approach to COVID-19 infection risk assessment in virtual visits: A case report. J Am Med Inform Assoc 2020; 27:1321-1325. [PMID: 32449766 PMCID: PMC7313981 DOI: 10.1093/jamia/ocaa105] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2020] [Revised: 05/07/2020] [Accepted: 05/21/2020] [Indexed: 12/15/2022] Open
Abstract
Objective In an effort to improve the efficiency of computer algorithms applied to screening for coronavirus disease 2019 (COVID-19) testing, we used natural language processing and artificial intelligence–based methods with unstructured patient data collected through telehealth visits. Materials and Methods After segmenting and parsing documents, we conducted analysis of overrepresented words in patient symptoms. We then developed a word embedding–based convolutional neural network for predicting COVID-19 test results based on patients’ self-reported symptoms. Results Text analytics revealed that concepts such as smell and taste were more prevalent than expected in patients testing positive. As a result, screening algorithms were adapted to include these symptoms. The deep learning model yielded an area under the receiver-operating characteristic curve of 0.729 for predicting positive results and was subsequently applied to prioritize testing appointment scheduling. Conclusions Informatics tools such as natural language processing and artificial intelligence methods can have significant clinical impacts when applied to data streams early in the development of clinical systems for outbreak response.
Collapse
Affiliation(s)
- Jihad S Obeid
- Department of Public Health Sciences, Medical University of South Carolina, Charleston, South Carolina, USA.,Biomedical Informatics Center, Medical University of South Carolina, Charleston, South Carolina, USA
| | - Matthew Davis
- Information Solutions, Medical University of South Carolina, Charleston, South Carolina, USA
| | - Matthew Turner
- Information Solutions, Medical University of South Carolina, Charleston, South Carolina, USA
| | - Stephane M Meystre
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, South Carolina, USA.,Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, Charleston, South Carolina, USA
| | - Paul M Heider
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, South Carolina, USA
| | - Edward C O'Bryan
- Department of Emergency Medicine, Medical University of South Carolina, Charleston, South Carolina, USA
| | - Leslie A Lenert
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, South Carolina, USA.,Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA
| |
Collapse
|
114
|
Agaronnik ND, Lindvall C, El-Jawahri A, He W, Iezzoni LI. Challenges of Developing a Natural Language Processing Method With Electronic Health Records to Identify Persons With Chronic Mobility Disability. Arch Phys Med Rehabil 2020; 101:1739-1746. [PMID: 32446905 PMCID: PMC7529728 DOI: 10.1016/j.apmr.2020.04.024] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2020] [Revised: 04/27/2020] [Accepted: 04/28/2020] [Indexed: 10/24/2022]
Abstract
OBJECTIVE To assess the utility of applying natural language processing (NLP) to electronic health records (EHRs) to identify individuals with chronic mobility disability. DESIGN We used EHRs from the Research Patient Data Repository, which contains EHRs from a large Massachusetts health care delivery system. This analysis was part of a larger study assessing the effects of disability on diagnosis of colorectal cancer. We applied NLP text extraction software to longitudinal EHRs of colorectal cancer patients to identify persons who use a wheelchair (our indicator of mobility disability for this analysis). We manually reviewed the clinical notes identified by NLP using directed content analysis to identify true cases using wheelchairs, duration or chronicity of use, and documentation quality. SETTING EHRs from large health care delivery system PARTICIPANTS: Patients (N=14,877) 21-75 years old who were newly diagnosed with colorectal cancer between 2005 and 2017. INTERVENTIONS Not applicable. MAIN OUTCOME MEASURES Confirmation of patients' chronic wheelchair use in NLP-flagged notes; quality of disability documentation. RESULTS We identified 14,877 patients with colorectal cancer with 303,182 associated clinical notes. NLP screening identified 1482 (0.5%) notes that contained 1+ wheelchair-associated keyword. These notes were associated with 420 patients (2.8% of colorectal cancer population). Of the 1482 notes, 286 (19.3%, representing 105 patients, 0.7% of the total) contained documentation of reason for wheelchair use and duration. Directed content analysis identified 3 themes concerning disability documentation: (1) wheelchair keywords used in specific EHR contexts; (2) reason for wheelchair not clearly stated; and (3) duration of wheelchair use not consistently documented. CONCLUSIONS NLP offers an option to screen for patients with chronic mobility disability in much less time than required by manual chart review. Nonetheless, manual chart review must confirm that flagged patients have chronic mobility disability (are not false positives). Notes, however, often have inadequate disability documentation.
Collapse
Affiliation(s)
- Nicole D Agaronnik
- Health Policy Research Center, Mongan Institute, Massachusetts General Hospital, Boston, MA
| | - Charlotta Lindvall
- Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, MA; Division of Palliative Medicine, Brigham and Women's Hospital, Boston, MA
| | - Areej El-Jawahri
- Department of Medicine, Harvard Medical School, Boston, MA; Division of Hematology and Oncology, Massachusetts General Hospital, Boston, MA
| | - Wei He
- Division of General Internal Medicine, Massachusetts General Hospital, Boston, MA
| | - Lisa I Iezzoni
- Health Policy Research Center, Mongan Institute, Massachusetts General Hospital, Boston, MA; Department of Medicine, Harvard Medical School, Boston, MA.
| |
Collapse
|
115
|
Miller HN, Charleston J, Wu B, Gleason K, White K, Dennison Himmelfarb CR, Ford DE, Plante TB, Gelber AC, Appel LJ, Miller ER, Juraschek SP. Use of electronic recruitment methods in a clinical trial of adults with gout. Clin Trials 2020; 18:92-103. [PMID: 32933342 DOI: 10.1177/1740774520956969] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
BACKGROUND/AIMS Electronic-based recruitment methods are increasingly utilized in clinical trials to recruit and enroll research participants. The cost-effectiveness of electronic-based methods and impact on sample generalizability is unknown. We compared recruitment yields, cost-effectiveness, and demographic characteristics across several electronic and traditional recruitment methods. METHODS We analyzed data from the diet gout trial recruitment campaign. The diet gout trial was a randomized, controlled, cross-over trial that examined the effects of a dietary approaches to stop hypertension (DASH)-like diet on uric acid levels in adults with gout. We used four electronic medical record and four non-electronic medical record-based recruitment methods to identify and recruit potentially eligible participants. We calculated the response rate, screening visit completion rate, and randomization rate for each method. We also determined cost per response, the screening, and randomization for each method. Finally, we compared the demographic characteristics among individuals who completed the screening visit by recruitment method. RESULTS Of the 294 adults who responded to the recruitment campaign, 51% were identified from electronic medical record-based methods. Patient portal messaging, an electronic medical record-based method, resulted in the highest response rate (4%), screening visit completion rate (37%), and randomization rate (21%) among these eight methods. Electronic medical record-based methods ($60) were more cost-effective per response than non-electronic medical record-based methods ($107). Electronic-based methods, including patient portal messaging and Facebook, had the highest proportion of White individuals screened (52% and 60%). Direct mail to non-active patient portal increased enrollment of traditionally under-represented groups, including both women and African Americans. CONCLUSION An electronic medical record-based recruitment strategy that utilized the electronic medical record for participant identification and postal mailing for participant outreach was cost-effective and increased participation of under-represented groups. This hybrid strategy represents a promising approach to improve the timely execution and broad generalizability of future clinical trials.
Collapse
Affiliation(s)
- Hailey N Miller
- School of Nursing, Johns Hopkins University, Baltimore, MD, USA.,Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA
| | - Jeanne Charleston
- Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA.,Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Beiwen Wu
- Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Kelly Gleason
- School of Nursing, Johns Hopkins University, Baltimore, MD, USA.,Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA
| | - Karen White
- Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Cheryl R Dennison Himmelfarb
- School of Nursing, Johns Hopkins University, Baltimore, MD, USA.,Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA
| | - Daniel E Ford
- Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA.,Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Timothy B Plante
- Department of Medicine, Larner College of Medicine, The University of Vermont, Burlington, VT, USA
| | - Allan C Gelber
- Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA.,Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Lawrence J Appel
- Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA.,Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA.,Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Edgar R Miller
- Institute for Clinical and Translational Research, Johns Hopkins University, Baltimore, MD, USA.,Welch Center for Prevention, Epidemiology, and Clinical Research, School of Medicine, Johns Hopkins University, Baltimore, MD, USA.,Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Stephen P Juraschek
- Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| |
Collapse
|
116
|
Abstract
Large registries, administrative data, and the electronic health record (EHR) offer opportunities to identify patients with heart failure, which can be used for research purposes, process improvement, and optimal care delivery. Identification of cases is challenging because of the heterogeneous nature of the disease, which encompasses various phenotypes that may respond differently to treatment. The increasing availability of both structured and unstructured data in the EHR has expanded opportunities for cohort construction. This article reviews the current literature on approaches to identification of heart failure, and looks toward the future of machine learning, big data, and phenomapping.
Collapse
Affiliation(s)
- Bernard S Kadosh
- Leon H. Charney Division of Cardiology, Department of Medicine, New York University School of Medicine, New York, NY, USA
| | - Stuart D Katz
- Leon H. Charney Division of Cardiology, Department of Medicine, New York University School of Medicine, New York, NY, USA
| | - Saul Blecker
- Department of Population Health, NYU School of Medicine, New York, NY, USA; Department of Medicine, NYU School of Medicine, New York, NY, USA; Center for Healthcare Innovation and Delivery Science, NYU Langone Health, New York, NY, USA.
| |
Collapse
|
117
|
Lhatoo SD, Bernasconi N, Blumcke I, Braun K, Buchhalter J, Denaxas S, Galanopoulou A, Josephson C, Kobow K, Lowenstein D, Ryvlin P, Schulze-Bonhage A, Sahoo SS, Thom M, Thurman D, Worrell G, Zhang GQ, Wiebe S. Big data in epilepsy: Clinical and research considerations. Report from the Epilepsy Big Data Task Force of the International League Against Epilepsy. Epilepsia 2020; 61:1869-1883. [PMID: 32767763 DOI: 10.1111/epi.16633] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2019] [Revised: 07/07/2020] [Accepted: 07/08/2020] [Indexed: 12/25/2022]
Abstract
Epilepsy is a heterogeneous condition with disparate etiologies and phenotypic and genotypic characteristics. Clinical and research aspects are accordingly varied, ranging from epidemiological to molecular, spanning clinical trials and outcomes, gene and drug discovery, imaging, electroencephalography, pathology, epilepsy surgery, digital technologies, and numerous others. Epilepsy data are collected in the terabytes and petabytes, pushing the limits of current capabilities. Modern computing firepower and advances in machine and deep learning, pioneered in other diseases, open up exciting possibilities for epilepsy too. However, without carefully designed approaches to acquiring, standardizing, curating, and making available such data, there is a risk of failure. Thus, careful construction of relevant ontologies, with intimate stakeholder inputs, provides the requisite scaffolding for more ambitious big data undertakings, such as an epilepsy data commons. In this review, we assess the clinical and research epilepsy landscapes in the big data arena, current challenges, and future directions, and make the case for a systematic approach to epilepsy big data.
Collapse
Affiliation(s)
- Samden D Lhatoo
- University of Texas Health Sciences Center at Houston, Houston, Texas
| | - Neda Bernasconi
- Neuroimaging of Epilepsy Laboratory, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Ingmar Blumcke
- Friedrich-Alexander University Erlangen-Nürnberg, University Hospital Erlangen, Erlangen, Germany
| | - Kees Braun
- Department of Child Neurology, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Jeffrey Buchhalter
- Department of Neurology, St Joseph's Hospital and Medical Center, Phoenix, Arizona
| | - Spiros Denaxas
- Institute of Health Informatics, University College London, London, UK
| | - Aristea Galanopoulou
- Saul Korey Department of Neurology, Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Bronx, New York
| | - Colin Josephson
- Department of Clinical Neurosciences, University of Calgary, Calgary, Canada
| | - Katja Kobow
- Friedrich-Alexander University Erlangen-Nürnberg, University Hospital Erlangen, Erlangen, Germany
| | - Daniel Lowenstein
- Department of Neurology, University of California, San Francisco, San Francisco, California
| | - Philippe Ryvlin
- Department of Neurosciences, University of Lausanne, Lausanne, Switzerland
| | | | - Satya S Sahoo
- Department of Population and Quantitative Health Sciences, School of Medicine, Case Western Reserve University, Cleveland, OH, USA
| | - Maria Thom
- Institute of Neurology, University College London, London, UK
| | | | - Greg Worrell
- Department of Neurology, Mayo Clinic, Rochester, Minnesota
| | - Guo-Qiang Zhang
- University of Texas Health Sciences Center at Houston, Houston, Texas
| | - Samuel Wiebe
- Department of Clinical Neurosciences, University of Calgary, Calgary, Canada
| |
Collapse
|
118
|
EHR phenotyping for research recruitment: Researcher, IRB, and physician perspectives on approaches to contacting patients. J Clin Transl Sci 2020; 5:e32. [PMID: 33948255 PMCID: PMC8057488 DOI: 10.1017/cts.2020.524] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/25/2022] Open
Abstract
Introduction: Failure to achieve accrual goals is a common problem in health-related research. Electronic health records represent a promising resource, offering the ability to identify a precisely defined cohort of patients who meet inclusion/exclusion criteria. However, challenges associated with the recruitment process remain and institutional policies vary. Methods: We interviewed researchers, institutional review board chairs, and primary care physicians in North Carolina and Tennessee. Questions focused on strategies for initiating contact with potentially eligible patients, as well as recruitment letters asking recipients to opt in versus opt out of further communication. Results: When we asked about initiating contact with prospective participants, qualitative themes included trust, credibility, and established relationships; research efficiency and validity; privacy and autonomy; the intersection between research and clinical care; and disruption to physician–researcher and physician–patient relationships. All interviewees said it was acceptable for researchers to contact patients through their physicians; most said it was acceptable for researchers to contact patients directly. Over half chose contact through physicians as more appropriate. Regarding recruitment letters, qualitative themes included the quality of the participant pool; privacy and control; research efficiency and representativeness; and patients’ opportunity to make their own decisions. All interviewees said asking recipients to opt in to further communication was acceptable; nearly all said opt out was acceptable. Similar proportions chose each approach as more appropriate. Conclusions: Comparing these results to our previous research with patients reveals potential differences in stakeholder perspectives. We offer suggestions for developing balanced approaches that respect patients and facilitate the advancement of science.
Collapse
|
119
|
Reducing Bias Due to Outcome Misclassification for Epidemiologic Studies Using EHR-derived Probabilistic Phenotypes. Epidemiology 2020; 31:542-550. [DOI: 10.1097/ede.0000000000001193] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
120
|
Venkataraman GR, Pineda AL, Bear Don’t Walk IV OJ, Zehnder AM, Ayyar S, Page RL, Bustamante CD, Rivas MA. FasTag: Automatic text classification of unstructured medical narratives. PLoS One 2020; 15:e0234647. [PMID: 32569327 PMCID: PMC7307763 DOI: 10.1371/journal.pone.0234647] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2020] [Accepted: 05/30/2020] [Indexed: 02/07/2023] Open
Abstract
Unstructured clinical narratives are continuously being recorded as part of delivery of care in electronic health records, and dedicated tagging staff spend considerable effort manually assigning clinical codes for billing purposes. Despite these efforts, however, label availability and accuracy are both suboptimal. In this retrospective study, we aimed to automate the assignment of top-level International Classification of Diseases version 9 (ICD-9) codes to clinical records from human and veterinary data stores using minimal manual labor and feature curation. Automating top-level annotations could in turn enable rapid cohort identification, especially in a veterinary setting. To this end, we trained long short-term memory (LSTM) recurrent neural networks (RNNs) on 52,722 human and 89,591 veterinary records. We investigated the accuracy of both separate-domain and combined-domain models and probed model portability. We established relevant baseline classification performances by training Decision Trees (DT) and Random Forests (RF). We also investigated whether transforming the data using MetaMap Lite, a clinical natural language processing tool, affected classification performance. We showed that the LSTM-RNNs accurately classify veterinary and human text narratives into top-level categories with an average weighted macro F1 score of 0.74 and 0.68 respectively. In the "neoplasia" category, the model trained on veterinary data had a high validation accuracy in veterinary data and moderate accuracy in human data, with F1 scores of 0.91 and 0.70 respectively. Our LSTM method scored slightly higher than that of the DT and RF models. The use of LSTM-RNN models represents a scalable structure that could prove useful in cohort identification for comparative oncology studies. Digitization of human and veterinary health information will continue to be a reality, particularly in the form of unstructured narratives. Our approach is a step forward for these two domains to learn from and inform one another.
Collapse
Affiliation(s)
- Guhan Ram Venkataraman
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, CA, United States of America
| | - Arturo Lopez Pineda
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, CA, United States of America
| | - Oliver J. Bear Don’t Walk IV
- Department of Biomedical Informatics, Vagelos College of Physicians and Surgeons, Columbia University, New York, NY, United States of America
| | | | - Sandeep Ayyar
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, CA, United States of America
| | - Rodney L. Page
- Department of Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Colorado State University, Fort Collins, CO, United States of America
| | - Carlos D. Bustamante
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, CA, United States of America
- Chan Zuckerberg Biohub, San Francisco, CA, United States of America
| | - Manuel A. Rivas
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, CA, United States of America
| |
Collapse
|
121
|
Montvida O, Dibato JE, Paul S. Evaluating the Representativeness of US Centricity Electronic Medical Records With Reports From the Centers for Disease Control and Prevention: Comparative Study on Office Visits and Cardiometabolic Conditions. JMIR Med Inform 2020; 8:e17174. [PMID: 32490850 PMCID: PMC7301254 DOI: 10.2196/17174] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2019] [Revised: 02/08/2020] [Accepted: 04/21/2020] [Indexed: 12/25/2022] Open
Abstract
Background Electronic medical record (EMR)–based clinical and epidemiological research has dramatically increased over the last decade, although establishing the generalizability of such big databases for conducting epidemiological studies has been an ongoing challenge. To draw meaningful inferences from such studies, it is essential to fully understand the characteristics of the underlying population and potential biases in EMRs. Objective This study aimed to assess the generalizability and representativity of the widely used US Centricity Electronic Medical Record (CEMR), a primary and ambulatory care EMR for population health research, using data from the National Ambulatory Medical Care Surveys (NAMCS) and the National Health and Nutrition Examination Surveys (NHANES). Methods The number of office visits reported in the NAMCS, designed to meet the need for objective and reliable information about the provision and the use of ambulatory medical care services, was compared with similar data from the CEMR. The distribution of major cardiometabolic diseases in the NHANES, designed to assess the health and nutritional status of adults and children in the United States, was compared with similar data from the CEMR. Results Gender and ethnicity distributions were similar between the NAMCS and the CEMR. Younger patients (aged <15 years) were underrepresented in the CEMR compared with the NAMCS. The number of office visits per 100 persons per year was similar: 277.9 (95% CI 259.3-296.5) in the NAMCS and 284.6 (95% CI 284.4-284.7) in the CEMR. However, the number of visits for males was significantly higher in the CEMR (CEMR: 270.8 and NAMCS: 239.0). West and South regions were underrepresented and overrepresented, respectively, in the CEMR. The overall prevalence of diabetes along with age and gender distribution was similar in the CEMR and the NHANES: overall prevalence, 10.1% and 9.7%; male, 11.5% and 10.8%; female, 9.1% and 8.8%; age 20 to 40 years, 2.5% and 1.8%; and age 40 to 60 years, 9.4% and 11.1%, respectively. The prevalence of obesity was similar: 42.1% and 39.6%, with similar age and female distribution (41.5% and 41.1%) but different male distribution (42.7% and 37.9%). The overall prevalence of high cholesterol along with age and female distribution was similar in the CEMR and the NHANES: overall prevalence, 12.4% and 12.4%; and female, 14.8% and 13.2%, respectively. The overall prevalence of hypertension was significantly higher in the CEMR (33.5%) than in the NHANES (95% CI: 27.0%-31.0%). Conclusions The distribution of major cardiometabolic diseases in the CEMR is comparable with the national survey results. The CEMR represents the general US population well in terms of office visits and major chronic conditions, whereas the potential subgroup differences in terms of age and gender distribution and prevalence may differ and, therefore, should be carefully taken care of in future studies.
Collapse
Affiliation(s)
- Olga Montvida
- Melbourne EpiCentre, University of Melbourne, Melbourne, Australia
| | - John Epoh Dibato
- Melbourne EpiCentre, University of Melbourne, Melbourne, Australia
| | - Sanjoy Paul
- Melbourne EpiCentre, University of Melbourne, Melbourne, Australia
| |
Collapse
|
122
|
Callahan A, Shah NH, Chen JH. Research and Reporting Considerations for Observational Studies Using Electronic Health Record Data. Ann Intern Med 2020; 172:S79-S84. [PMID: 32479175 PMCID: PMC7413106 DOI: 10.7326/m19-0873] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/16/2023] Open
Abstract
Electronic health records (EHRs) are an increasingly important source of real-world health care data for observational research. Analyses of data collected for purposes other than research require careful consideration of data quality as well as the general research and reporting principles relevant to observational studies. The core principles for observational research in general also apply to observational research using EHR data, and these are well addressed in prior literature and guidelines. This article provides additional recommendations for EHR-based research. Considerations unique to EHR-based studies include assessment of the accuracy of computer-executable cohort definitions that can incorporate unstructured data from clinical notes and management of data challenges, such as irregular sampling, missingness, and variation across time and place. Principled application of existing research and reporting guidelines alongside these additional considerations will improve the quality of EHR-based observational studies.
Collapse
Affiliation(s)
- Alison Callahan
- Center for Biomedical Informatics Research, School of Medicine, Stanford University (A.C., N.H.S.)
| | - Nigam H Shah
- Center for Biomedical Informatics Research, School of Medicine, Stanford University (A.C., N.H.S.)
| | - Jonathan H Chen
- Division of Hospital Medicine, School of Medicine, Stanford University (J.H.C.)
| |
Collapse
|
123
|
McGuirl MR, Smith SP, Sandstede B, Ramachandran S. Detecting Shared Genetic Architecture Among Multiple Phenotypes by Hierarchical Clustering of Gene-Level Association Statistics. Genetics 2020; 215:511-529. [PMID: 32245788 PMCID: PMC7268989 DOI: 10.1534/genetics.120.303096] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2020] [Accepted: 03/31/2020] [Indexed: 12/31/2022] Open
Abstract
Emerging large-scale biobanks pairing genotype data with phenotype data present new opportunities to prioritize shared genetic associations across multiple phenotypes for molecular validation. Past research, by our group and others, has shown gene-level tests of association produce biologically interpretable characterization of the genetic architecture of a given phenotype. Here, we present a new method, Ward clustering to identify Internal Node branch length outliers using Gene Scores (WINGS), for identifying shared genetic architecture among multiple phenotypes. The objective of WINGS is to identify groups of phenotypes, or "clusters," sharing a core set of genes enriched for mutations in cases. We validate WINGS using extensive simulation studies and then combine gene-level association tests with WINGS to identify shared genetic architecture among 81 case-control and seven quantitative phenotypes in 349,468 European-ancestry individuals from the UK Biobank. We identify eight prioritized phenotype clusters and recover multiple published gene-level associations within prioritized clusters.
Collapse
Affiliation(s)
- Melissa R McGuirl
- Division of Applied Mathematics, Brown University, Providence, Rhode Island 02912
| | - Samuel Pattillo Smith
- Center for Computational Molecular Biology, Brown University, Providence, Rhode Island 02912
- Department of Ecology and Evolutionary Biology, Brown University, Providence, Rhode Island 02912
| | - Björn Sandstede
- Division of Applied Mathematics, Brown University, Providence, Rhode Island 02912
- Data Science Initiative, Brown University, Providence, Rhode Island 02912
| | - Sohini Ramachandran
- Center for Computational Molecular Biology, Brown University, Providence, Rhode Island 02912
- Department of Ecology and Evolutionary Biology, Brown University, Providence, Rhode Island 02912
| |
Collapse
|
124
|
Heider PM, Obeid JS, Meystre SM. A Comparative Analysis of Speed and Accuracy for Three Off-the-Shelf De-Identification Tools. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE PROCEEDINGS. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE 2020; 2020:241-250. [PMID: 32477643 PMCID: PMC7233098] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
A growing quantity of health data is being stored in Electronic Health Records (EHR). The free-text section of these clinical notes contains important patient and treatment information for research but also contains Personally Identifiable Information (PII), which cannot be freely shared within the research community without compromising patient confidentiality and privacy rights. Significant work has been invested in investigating automated approaches to text de-identification, the process of removing or redacting PII. Few studies have examined the performance of existing de-identification pipelines in a controlled comparative analysis. In this study, we use publicly available corpora to analyze speed and accuracy differences between three de-identification systems that can be run off-the-shelf: Amazon Comprehend Medical PHId, Clinacuity's CliniDeID, and the National Library of Medicine's Scrubber. No single system dominated all the compared metrics. NLM Scrubber was the fastest while CliniDeID generally had the highest accuracy.
Collapse
Affiliation(s)
- Paul M Heider
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC
| | - Jihad S Obeid
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC
| | - Stéphane M Meystre
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC
| |
Collapse
|
125
|
Dai HJ, Wang FD, Chen CW, Su CH, Wu CS, Jonnagaddala J. Cohort selection for clinical trials using multiple instance learning. J Biomed Inform 2020; 107:103438. [PMID: 32360937 DOI: 10.1016/j.jbi.2020.103438] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2019] [Revised: 02/29/2020] [Accepted: 04/27/2020] [Indexed: 10/24/2022]
Abstract
Identifying patients eligible for clinical trials using electronic health records (EHRs) is a challenging task usually requiring a comprehensive analysis of information stored in multiple EHRs of a patient. The goal of this study is to investigate different methods and their effectiveness in identifying patients that meet specific eligibility selection criteria based on patients' longitudinal records. An unstructured dataset released by the n2c2 cohort selection for clinical trials track was used, each of which included 2-5 records manually annotated to thirteen pre-defined selection criteria. Unlike the other studies, we formulated the problem as a multiple instance learning (MIL) task and compared the performance with that of the rule-based and the single instance-based classifiers. Our official best run achieved an average micro-F score of 0.8765 which was ranked as one of the top ten results in the track. Further experiments demonstrated that the performance of the MIL-based classifiers consistently yield better performance than their single-instance counterparts in the criteria that require the overall comprehension of the information distributed among all of the patient's EHRs. Rule-based and single instance learning approaches exhibited better performance in criteria that don't require a consideration of several factors across records. This study demonstrated that cohort selection using longitudinal patient records can be formulated as a MIL problem. Our results exhibit that the MIL-based classifiers supplement the rule-based methods and provide better results in comparison to the single instance learning approaches.
Collapse
Affiliation(s)
- Hong-Jie Dai
- Department of Electrical Engineering, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan, ROC; School of Post-Baccalaureate Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan, ROC; National Institute of Cancer Research, National Health Research Institutes, Taiwan, R.O.C..
| | - Feng-Duo Wang
- Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan, ROC
| | - Chih-Wei Chen
- Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei, Taiwan, ROC
| | - Chu-Hsien Su
- Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan, ROC
| | - Chi-Shin Wu
- Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan, ROC
| | | |
Collapse
|
126
|
To D, Sharma B, Karnik N, Joyce C, Dligach D, Afshar M. Validation of an alcohol misuse classifier in hospitalized patients. Alcohol 2020; 84:49-55. [PMID: 31574300 DOI: 10.1016/j.alcohol.2019.09.008] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2019] [Revised: 09/10/2019] [Accepted: 09/25/2019] [Indexed: 12/21/2022]
Abstract
BACKGROUND Current modes of identifying alcohol misuse in hospitalized patients rely on self-report questionnaires and diagnostic codes that have limitations, including low sensitivity. Information in the clinical notes of the electronic health record (EHR) may further augment the identification of alcohol misuse. Natural language processing (NLP) with supervised machine learning has been successful at analyzing clinical notes and identifying cases of alcohol misuse in trauma patients. METHODS An alcohol misuse NLP classifier, previously developed on trauma patients who completed the Alcohol Use Disorders Identification Test, was validated in a cohort of 1000 hospitalized patients at a large, tertiary health system between January 1, 2007 and September 1, 2017. The clinical notes were processed using the clinical Text Analysis and Knowledge Extraction System. The National Institute on Alcohol Abuse and Alcoholism (NIAAA) guidelines for alcohol misuse were used during annotation of the medical records in our validation dataset. RESULTS The alcohol misuse classifier had an area under the receiver operating characteristic curve of 0.91 (95% CI 0.90-0.93) in the cohort of hospitalized patients. The sensitivity, specificity, positive predictive value, and negative predictive value were 0.88 (95% CI 0.85-0.90), 0.78 (95% CI 0.74-0.82), 0.85 (95% CI 0.82-0.87), and 0.82 (95% CI 0.78-0.86), respectively. The Hosmer-Lemeshow Test (p = 0.13) demonstrates good model fit. Additionally, there was a dose-dependent response in alcohol consumption behaviors across increasing strata of predicted probabilities for alcohol misuse. CONCLUSION The alcohol misuse NLP classifier had good discrimination and test characteristics in hospitalized patients. An approach using the clinical notes with NLP and supervised machine learning may better identify alcohol misuse cases than conventional methods solely relying on billing diagnostic codes.
Collapse
Affiliation(s)
- Daniel To
- Stritch School of Medicine, Loyola University Chicago, Maywood, IL, USA
| | - Brihat Sharma
- Department of Computer Science, Loyola University Chicago, Chicago, IL, USA
| | - Niranjan Karnik
- Department of Psychiatry & Behavioral Sciences, Rush Medical College, Rush University, Chicago, IL, USA
| | - Cara Joyce
- Stritch School of Medicine, Loyola University Chicago, Maywood, IL, USA; Department of Public Health Sciences, Loyola University Chicago, Maywood, IL, USA; Center for Health Outcomes and Informatics Research, Loyola University Chicago, Maywood, IL, USA
| | - Dmitriy Dligach
- Department of Computer Science, Loyola University Chicago, Chicago, IL, USA; Department of Public Health Sciences, Loyola University Chicago, Maywood, IL, USA; Center for Health Outcomes and Informatics Research, Loyola University Chicago, Maywood, IL, USA
| | - Majid Afshar
- Stritch School of Medicine, Loyola University Chicago, Maywood, IL, USA; Department of Public Health Sciences, Loyola University Chicago, Maywood, IL, USA; Center for Health Outcomes and Informatics Research, Loyola University Chicago, Maywood, IL, USA.
| |
Collapse
|
127
|
Sharma B, Dligach D, Swope K, Salisbury-Afshar E, Karnik NS, Joyce C, Afshar M. Publicly available machine learning models for identifying opioid misuse from the clinical notes of hospitalized patients. BMC Med Inform Decis Mak 2020; 20:79. [PMID: 32349766 PMCID: PMC7191715 DOI: 10.1186/s12911-020-1099-y] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2019] [Accepted: 04/22/2020] [Indexed: 12/15/2022] Open
Abstract
Background Automated de-identification methods for removing protected health information (PHI) from the source notes of the electronic health record (EHR) rely on building systems to recognize mentions of PHI in text, but they remain inadequate at ensuring perfect PHI removal. As an alternative to relying on de-identification systems, we propose the following solutions: (1) Mapping the corpus of documents to standardized medical vocabulary (concept unique identifier [CUI] codes mapped from the Unified Medical Language System) thus eliminating PHI as inputs to a machine learning model; and (2) training character-based machine learning models that obviate the need for a dictionary containing input words/n-grams. We aim to test the performance of models with and without PHI in a use-case for an opioid misuse classifier. Methods An observational cohort sampled from adult hospital inpatient encounters at a health system between 2007 and 2017. A case-control stratified sampling (n = 1000) was performed to build an annotated dataset for a reference standard of cases and non-cases of opioid misuse. Models for training and testing included CUI codes, character-based, and n-gram features. Models applied were machine learning with neural network and logistic regression as well as expert consensus with a rule-based model for opioid misuse. The area under the receiver operating characteristic curves (AUROC) were compared between models for discrimination. The Hosmer-Lemeshow test and visual plots measured model fit and calibration. Results Machine learning models with CUI codes performed similarly to n-gram models with PHI. The top performing models with AUROCs > 0.90 included CUI codes as inputs to a convolutional neural network, max pooling network, and logistic regression model. The top calibrated models with the best model fit were the CUI-based convolutional neural network and max pooling network. The top weighted CUI codes in logistic regression has the related terms ‘Heroin’ and ‘Victim of abuse’. Conclusions We demonstrate good test characteristics for an opioid misuse computable phenotype that is void of any PHI and performs similarly to models that use PHI. Herein we share a PHI-free, trained opioid misuse classifier for other researchers and health systems to use and benchmark to overcome privacy and security concerns.
Collapse
Affiliation(s)
- Brihat Sharma
- Department of Computer Science, Loyola University Chicago, Chicago, IL, USA
| | - Dmitriy Dligach
- Department of Computer Science, Loyola University Chicago, Chicago, IL, USA.,Center for Health Outcomes and Informatics Research, Loyola University Chicago, 2160 S. First Avenue, Maywood, IL, 60156, USA
| | - Kristin Swope
- Stritch School of Medicine, Loyola University Chicago, Maywood, IL, USA
| | - Elizabeth Salisbury-Afshar
- Center for Multi-System Solutions to the Opioid Epidemic, American Institute for Research, Chicago, IL, USA
| | - Niranjan S Karnik
- Department of Psychiatry, Rush University Medical Center, Chicago, IL, USA
| | - Cara Joyce
- Center for Health Outcomes and Informatics Research, Loyola University Chicago, 2160 S. First Avenue, Maywood, IL, 60156, USA.,Stritch School of Medicine, Loyola University Chicago, Maywood, IL, USA
| | - Majid Afshar
- Center for Health Outcomes and Informatics Research, Loyola University Chicago, 2160 S. First Avenue, Maywood, IL, 60156, USA. .,Department of Health Informatics and Data Science, Loyola University Chicago, Maywood, IL, USA. .,Department of Medicine, Loyola University Medical Center, Maywood, IL, USA.
| |
Collapse
|
128
|
Floris-Moore M, Edmonds A, Napravnik S, Adimora AA. Computerized Adjudication of Coronary Heart Disease Events Using the Electronic Medical Record in HIV Clinical Research: Possibilities and Challenges Ahead. AIDS Res Hum Retroviruses 2020; 36:306-313. [PMID: 31407587 DOI: 10.1089/aid.2019.0036] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
This pilot study assessed feasibility of computer-assisted electronic medical record (EMR) abstraction to ascertain coronary heart disease (CHD) event hospitalizations. We included a sample of 87 hospitalization records from participants the University of North Carolina (UNC) site of the Women's Interagency HIV Study (WIHS) and UNC Center for AIDS Research (CFAR) HIV Clinical Cohort who were hospitalized within UNC Healthcare System from July 2004 to July 2015. We compared a computer algorithm utilizing diagnosis/procedure codes, medications, and cardiac enzyme levels to adjudicate CHD events [myocardial infarction (MI)/coronary revascularization] from the EMR to standardized manual chart adjudication. Of 87 hospitalizations, 42 were classified as definite, 25 probable, and 20 non-CHD events by manual chart adjudication. A computer algorithm requiring presence of ≥1 CHD-related International Classification of Diseases, 9th Revision (ICD-9)/Current Procedural Terminology (CPT) code correctly identified 24 of 42 definite (57%), 29 of 67 probable/definite CHD (43%), and 95% of non-CHD events; additionally requiring clinically defined cardiac enzyme levels or administration of MI-related medications correctly identified 55%, 42%, and 95% of such events, respectively. Requiring any one of the ICD-9/CPT or cardiac enzyme criteria correctly identified 98% of definite, 97% of probable/definite CHD, and 85% of non-CHD events. Challenges included difficulty matching hospitalization dates, incomplete diagnosis code data, and multiple field names/locations of laboratory/medication data. Computer algorithms comprising only ICD-9/CPT codes failed to identify a sizable proportion of CHD events. Using a less restrictive algorithm yielded fewer missed events but increased the false-positive rate. Despite potential benefits of EMR-based research, there remain several challenges to fully computerized adjudication of CHD events.
Collapse
Affiliation(s)
- Michelle Floris-Moore
- Division of Infectious Diseases, Department of Medicine, School of Medicine, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
- Department of Medicine, Center for AIDS Research, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| | - Andrew Edmonds
- Department of Epidemiology, Gillings School of Global Public Health, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| | - Sonia Napravnik
- Division of Infectious Diseases, Department of Medicine, School of Medicine, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
- Department of Medicine, Center for AIDS Research, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| | - Adaora A. Adimora
- Division of Infectious Diseases, Department of Medicine, School of Medicine, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
- Department of Medicine, Center for AIDS Research, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
- Department of Epidemiology, Gillings School of Global Public Health, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| |
Collapse
|
129
|
Rasmussen LV, Brandt PS, Jiang G, Kiefer RC, Pacheco JA, Adekkanattu P, Ancker JS, Wang F, Xu Z, Pathak J, Luo Y. Considerations for Improving the Portability of Electronic Health Record-Based Phenotype Algorithms. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2020; 2019:755-764. [PMID: 32308871 PMCID: PMC7153055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
With the increased adoption of electronic health records, data collected for routine clinical care is used for health outcomes and population sciences research, including the identification of phenotypes. In recent years, research networks, such as eMERGE, OHDSI and PCORnet, have been able to increase statistical power and population diversity by combining patient cohorts. These networks share phenotype algorithms that are executed at each participating site. Here we observe experiences with phenotype algorithm portability across seven research networks and propose a generalizable framework for phenotype algorithm portability. Several strategies exist to increase the portability of phenotype algorithms, reducing the implementation effort needed by each site. These include using a common data model, standardized representation of the phenotype algorithm logic, and technical solutions to facilitate federated execution of queries. Portability is achieved by tradeoffs across three domains: Data, Authoring and Implementation, and multiple approaches were observed in representing portable phenotype algorithms. Our proposed framework will help guide future research in operationalizing phenotype algorithm portability at scale.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | - Fei Wang
- Northwestern University, Chicago, IL
| | | | | | - Yuan Luo
- Northwestern University, Chicago, IL
| |
Collapse
|
130
|
Ferrão JC, Oliveira MD, Janela F, Martins HMG, Gartner D. Can structured EHR data support clinical coding? A data mining approach. Health Syst (Basingstoke) 2020; 10:138-161. [PMID: 34104432 PMCID: PMC8143604 DOI: 10.1080/20476965.2020.1729666] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2019] [Accepted: 10/22/2019] [Indexed: 10/24/2022] Open
Abstract
Structured data formats are gaining momentum in electronic health records and can be leveraged for decision support and research. Nevertheless, such structured data formats have not been explored for clinical coding, which is an essential process requiring significant manual workload in health organisations. This article explores the extent to which fully structured clinical data can support assignment of clinical codes to inpatient episodes, through a methodology that tackles high dimensionality issues, addresses the multi-label nature of coding and optimises model parameters. The methodology encompasses transformation of raw data to define a feature set, build a data matrix representation, and testing combinations of feature selection methods with machine learning models to predict code assignment. The methodology was tested with a real hospital dataset and showed varying predictive power across codes, while demonstrating the potential of leveraging structuring data to reduce workload and increase efficiency in clinical coding.
Collapse
Affiliation(s)
- José Carlos Ferrão
- CEG-IST, Centre for Management Studies of Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal
| | - Mónica Duarte Oliveira
- CEG-IST, Centre for Management Studies of Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal
| | - Filipe Janela
- Investigação, Desenvolvimento e Inovação, SIEMENS Healthineers, Amadora, Portugal
| | - Henrique M. G. Martins
- Centre for Research and Creativity in Informatics (CI), Hospital Prof. Doutor Fernando Fonseca, Amadora, Portugal
| | | |
Collapse
|
131
|
Foster M, Albanese C, Chen Q, Sethares KA, Evans S, Lehmann LS, Spencer J, Joseph J. Heart Failure Dashboard Design and Validation to Improve Care of Veterans. Appl Clin Inform 2020; 11:153-159. [PMID: 32102107 DOI: 10.1055/s-0040-1701257] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Early electronic identification of patients at the highest risk for heart failure (HF) readmission presents a challenge. Data needed to identify HF patients are in a variety of areas in the electronic medical record (EMR) and in different formats. OBJECTIVE The purpose of this paper is to describe the development and data validation of a HF dashboard that monitors the overall metrics of outcomes and treatments of the veteran patient population with HF and enhancing the use of guideline-directed pharmacologic therapies. METHODS We constructed a dashboard that included several data points: care assessment need score; ejection fraction (EF); medication concordance; laboratory tests; history of HF; and specified comorbidities based on International Classification of Disease (ICD), ninth and tenth codes. Data validation testing with user test scripts was utilized to ensure output accuracy of the dashboard. Nine providers and key senior management participated in data validation. RESULTS A total of 43 medical records were reviewed and 66 HF dashboard data discrepancies were identified during development. Discrepancies identified included: generation of multiple EF values on a few patients, missing or incorrect ICD codes, laboratory omission, incorrect medication issue dates, patients incorrectly noted as nonconcordant for medications, and incorrect dates of last cardiology appointments. Continuous integration and builds identified defects-an important process of the verification and validation of biomedical software. Data validation and technical limitations are some challenges that were encountered during dashboard development. Evaluations by testers and their focused feedback contributed to the lessons learned from the challenges. CONCLUSION Continuous refinement with input from multiple levels of stakeholders is crucial to development of clinically useful dashboards. Extraction of all relevant information from EMRs, including the use of natural language processing, is crucial to development of dashboards that will help improve care of individual patients and populations.
Collapse
Affiliation(s)
- Marva Foster
- Emergency Services, VA Boston Healthcare System, Boston, Massachusetts, United States
| | - Catherine Albanese
- Data Management Office, VA New England Healthcare System, Bedford, Massachusetts, United States
| | - Qiang Chen
- Data Management Office, VA New England Healthcare System, Bedford, Massachusetts, United States
| | - Kristen A Sethares
- College of Nursing and Health Sciences, University of Massachusetts, Dartmouth, Massachusetts, United States
| | - Stewart Evans
- Data Management Office, VA New England Healthcare System, Bedford, Massachusetts, United States
| | - Lisa Soleymani Lehmann
- Department of Medicine, Brigham & Women's Hospital, Boston, Massachusetts, United States.,Department of Medicine, Harvard Medical School, Boston, Massachusetts, United States.,Department of Medicine, VA Boston Healthcare System, Boston, Massachusetts, United States
| | - Jacqueline Spencer
- Primary Care, VA New England Healthcare System, Bedford, Massachusetts, United States
| | - Jacob Joseph
- Department of Medicine, Brigham & Women's Hospital, Boston, Massachusetts, United States.,Department of Cardiology, VA Boston Healthcare System, Boston, Massachusetts, United States
| |
Collapse
|
132
|
Vos RA, Katayama T, Mishima H, Kawano S, Kawashima S, Kim JD, Moriya Y, Tokimatsu T, Yamaguchi A, Yamamoto Y, Wu H, Amstutz P, Antezana E, Aoki NP, Arakawa K, Bolleman JT, Bolton E, Bonnal RJP, Bono H, Burger K, Chiba H, Cohen KB, Deutsch EW, Fernández-Breis JT, Fu G, Fujisawa T, Fukushima A, García A, Goto N, Groza T, Hercus C, Hoehndorf R, Itaya K, Juty N, Kawashima T, Kim JH, Kinjo AR, Kotera M, Kozaki K, Kumagai S, Kushida T, Lütteke T, Matsubara M, Miyamoto J, Mohsen A, Mori H, Naito Y, Nakazato T, Nguyen-Xuan J, Nishida K, Nishida N, Nishide H, Ogishima S, Ohta T, Okuda S, Paten B, Perret JL, Prathipati P, Prins P, Queralt-Rosinach N, Shinmachi D, Suzuki S, Tabata T, Takatsuki T, Taylor K, Thompson M, Uchiyama I, Vieira B, Wei CH, Wilkinson M, Yamada I, Yamanaka R, Yoshitake K, Yoshizawa AC, Dumontier M, Kosaki K, Takagi T. BioHackathon 2015: Semantics of data for life sciences and reproducible research. F1000Res 2020; 9:136. [PMID: 32308977 PMCID: PMC7141167 DOI: 10.12688/f1000research.18236.1] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 02/05/2020] [Indexed: 01/08/2023] Open
Abstract
We report on the activities of the 2015 edition of the BioHackathon, an annual event that brings together researchers and developers from around the world to develop tools and technologies that promote the reusability of biological data. We discuss issues surrounding the representation, publication, integration, mining and reuse of biological data and metadata across a wide range of biomedical data types of relevance for the life sciences, including chemistry, genotypes and phenotypes, orthology and phylogeny, proteomics, genomics, glycomics, and metabolomics. We describe our progress to address ongoing challenges to the reusability and reproducibility of research results, and identify outstanding issues that continue to impede the progress of bioinformatics research. We share our perspective on the state of the art, continued challenges, and goals for future research and development for the life sciences Semantic Web.
Collapse
Affiliation(s)
- Rutger A. Vos
- Institute of Biology Leiden, Leiden University, Leiden, The Netherlands
- Naturalis Biodiversity Center, Leiden, The Netherlands
| | | | - Hiroyuki Mishima
- Department of Human Genetics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan
| | - Shin Kawano
- Database Center for Life Science, Tokyo, Japan
| | | | | | - Yuki Moriya
- Database Center for Life Science, Tokyo, Japan
| | | | | | | | - Hongyan Wu
- Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | | | - Erick Antezana
- Department of Biology, Norwegian University of Science and Technology, Trondheim, Norway
| | - Nobuyuki P. Aoki
- Faculty of Science and Engineering, SOKA University, Tokyo, Japan
| | - Kazuharu Arakawa
- Institute for Advanced Biosciences, Keio University, Tokyo, Japan
| | - Jerven T. Bolleman
- SIB Swiss Institute of Bioinformatics, Centre Medical Universitaire, Lausanne, Switzerland
| | - Evan Bolton
- National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, USA
| | - Raoul J. P. Bonnal
- Istituto Nazionale Genetica Molecolare, Romeo ed Enrica Invernizzi, Milan, Italy
| | | | - Kees Burger
- Dutch Techcentre for Life Sciences, Utrecht, The Netherlands
| | - Hirokazu Chiba
- National Institute for Basic Biology, National Institutes of Natural Sciences, Okazaki, Japan
| | - Kevin B. Cohen
- Computational Bioscience Program, University of Colorado School of Medicine, Denver, USA
- Université Paris-Saclay, LIMSI, CNRS, Paris, France
| | | | | | - Gang Fu
- National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, USA
| | | | | | | | - Naohisa Goto
- Research Institute for Microbial Diseases, Osaka University, Osaka, Japan
| | - Tudor Groza
- St Vincent's Clinical School, Faculty of Medicine, University of New South Wales, Darlinghurst, Australia
- Kinghorn Centre for Clinical Genomics, Garvan Institute of Medical Research, Darlinghurst, Australia
| | - Colin Hercus
- Novocraft Technologies Sdn. Bhd., Selangor, Malaysia
| | - Robert Hoehndorf
- Computational Bioscience Research Center, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
| | - Kotone Itaya
- Institute for Advanced Biosciences, Keio University, Tokyo, Japan
| | - Nick Juty
- European Molecular Biology Laboratory, European Bioinformatics Institute, Hinxton, UK
| | | | - Jee-Hyub Kim
- European Molecular Biology Laboratory, European Bioinformatics Institute, Hinxton, UK
| | - Akira R. Kinjo
- Institute for Protein Research, Osaka University, Osaka, Japan
| | - Masaaki Kotera
- School of Life Science and Technology, Tokyo Institute of Technology, Tokyo, Japan
| | - Kouji Kozaki
- The Institute of Scientific and Industrial Research, Osaka University, Osaka, Japan
| | | | - Tatsuya Kushida
- National Bioscience Database Center, Japan Science and Technology Agency, Tokyo, Japan
| | - Thomas Lütteke
- Institute of Veterinary Physiology and Biochemistry, Justus-Liebig University Giessen, Giessen, Germany
- Gesellschaft für innovative Personalwirtschaftssysteme mbH (GIP GmbH), Offenbach, Germany
| | | | | | - Attayeb Mohsen
- National Institutes of Biomedical Innovation, Health and Nutrition, Osaka, Japan
| | - Hiroshi Mori
- Center for Information Biology, National Institute of Genetics, Mishima, Japan
| | - Yuki Naito
- Database Center for Life Science, Tokyo, Japan
| | | | | | | | - Naoki Nishida
- Department of Systems Science, Osaka University, Osaka, Japan
| | - Hiroyo Nishide
- National Institute for Basic Biology, National Institutes of Natural Sciences, Okazaki, Japan
| | - Soichi Ogishima
- Tohoku Medical Megabank Organization, Tohoku University, Sendai, Japan
| | - Tazro Ohta
- Database Center for Life Science, Tokyo, Japan
| | - Shujiro Okuda
- Niigata University Graduate School of Medical and Dental Sciences, Niigata, Japan
| | - Benedict Paten
- UC Santa Cruz Genomics Institute, University of California, Santa Cruz, USA
| | | | - Philip Prathipati
- National Institutes of Biomedical Innovation, Health and Nutrition, Osaka, Japan
| | - Pjotr Prins
- University Medical Center Utrecht, Utrecht, The Netherlands
- University of Tennessee Health Science Center, Memphis, USA
| | - Núria Queralt-Rosinach
- Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA
| | | | - Shinya Suzuki
- School of Life Science and Technology, Tokyo Institute of Technology, Tokyo, Japan
| | - Tsuyosi Tabata
- Graduate School of Pharmaceutical Sciences, Kyoto University, Kyoto, Japan
| | | | - Kieron Taylor
- European Molecular Biology Laboratory, European Bioinformatics Institute, Hinxton, UK
| | - Mark Thompson
- Leiden University Medical Center, Leiden, The Netherlands
| | - Ikuo Uchiyama
- National Institute for Basic Biology, National Institutes of Natural Sciences, Okazaki, Japan
| | - Bruno Vieira
- WurmLab, School of Biological & Chemical Sciences, Queen Mary University of London, London, UK
| | - Chih-Hsuan Wei
- National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, USA
| | - Mark Wilkinson
- Escuela Técnica Superior de Ingeniería Agronómica, Alimentaria y de Biosistemas, Universidad Politécnica de Madrid, Madrid, Spain
| | | | | | - Kazutoshi Yoshitake
- Graduate School of Agricultural and Life Sciences, The University of Tokyo, Tokyo, Japan
| | | | - Michel Dumontier
- Institute of Data Science, Maastricht University, Maastricht, The Netherlands
| | - Kenjiro Kosaki
- Center for Medical Genetics, Keio University School of Medicine, Tokyo, Japan
| | - Toshihisa Takagi
- National Bioscience Database Center, Japan Science and Technology Agency, Tokyo, Japan
- Department of Biological Sciences, Graduate School of Science, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
133
|
Juhn Y, Liu H. Artificial intelligence approaches using natural language processing to advance EHR-based clinical research. J Allergy Clin Immunol 2020; 145:463-469. [PMID: 31883846 PMCID: PMC7771189 DOI: 10.1016/j.jaci.2019.12.897] [Citation(s) in RCA: 93] [Impact Index Per Article: 23.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2019] [Revised: 12/18/2019] [Accepted: 12/19/2019] [Indexed: 01/17/2023]
Abstract
The wide adoption of electronic health record systems in health care generates big real-world data that open new venues to conduct clinical research. As a large amount of valuable clinical information is locked in clinical narratives, natural language processing techniques as an artificial intelligence approach have been leveraged to extract information from clinical narratives in electronic health records. This capability of natural language processing potentially enables automated chart review for identifying patients with distinctive clinical characteristics in clinical care and reduces methodological heterogeneity in defining phenotype, obscuring biological heterogeneity in research concerning allergy, asthma, and immunology. This brief review discusses the current literature on the secondary use of electronic health record data for clinical research concerning allergy, asthma, and immunology and highlights the potential, challenges, and implications of natural language processing techniques.
Collapse
Affiliation(s)
- Young Juhn
- Precision Population Science Lab, Division of Community Pediatric and Adolescent Medicine, Department of Pediatric and Adolescent Medicine, Rochester, Minn; Division of Allergy, Department of Medicine, Mayo Clinic, Rochester, Minn.
| | - Hongfang Liu
- Division of Digital Health, Department of Health Sciences Research, Mayo Clinic, Rochester, Minn
| |
Collapse
|
134
|
Pfaff ER, Crosskey M, Morton K, Krishnamurthy A. Clinical Annotation Research Kit (CLARK): Computable Phenotyping Using Machine Learning. JMIR Med Inform 2020; 8:e16042. [PMID: 32012059 PMCID: PMC7007592 DOI: 10.2196/16042] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2019] [Revised: 10/30/2019] [Accepted: 12/16/2019] [Indexed: 01/02/2023] Open
Abstract
Computable phenotypes are algorithms that translate clinical features into code that can be run against electronic health record (EHR) data to define patient cohorts. However, computable phenotypes that only make use of structured EHR data do not capture the full richness of a patient’s medical record. While natural language processing (NLP) methods have shown success in extracting clinical features from text, the use of such tools has generally been limited to research groups with substantial NLP expertise. Our goal was to develop an open-source phenotyping software, Clinical Annotation Research Kit (CLARK), that would enable clinical and translational researchers to use machine learning–based NLP for computable phenotyping without requiring deep informatics expertise. CLARK enables nonexpert users to mine text using machine learning classifiers by specifying features for the software to match in clinical notes. Once the features are defined, the user-friendly CLARK interface allows the user to choose from a variety of standard machine learning algorithms (linear support vector machine, Gaussian Naïve Bayes, decision tree, and random forest), cross-validation methods, and the number of folds (cross-validation splits) to be used in evaluation of the classifier. Example phenotypes where CLARK has been applied include pediatric diabetes (sensitivity=0.91; specificity=0.98), symptomatic uterine fibroids (positive predictive value=0.81; negative predictive value=0.54), nonalcoholic fatty liver disease (sensitivity=0.90; specificity=0.94), and primary ciliary dyskinesia (sensitivity=0.88; specificity=1.0). In each of these use cases, CLARK allowed investigators to incorporate variables into their phenotype algorithm that would not be available as structured data. Moreover, the fact that nonexpert users can get started with machine learning–based NLP with limited informatics involvement is a significant improvement over the status quo. We hope to disseminate CLARK to other organizations that may not have NLP or machine learning specialists available, enabling wider use of these methods.
Collapse
Affiliation(s)
- Emily R Pfaff
- North Carolina Translational and Clinical Sciences Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | | | | | - Ashok Krishnamurthy
- Renaissance Computing Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| |
Collapse
|
135
|
Automated Assessment of Existing Patient's Revised Cardiac Risk Index Using Algorithmic Software. Anesth Analg 2020; 128:909-916. [PMID: 29847379 DOI: 10.1213/ane.0000000000003440] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
BACKGROUND Previous work in the field of medical informatics has shown that rules-based algorithms can be created to identify patients with various medical conditions; however, these techniques have not been compared to actual clinician notes nor has the ability to predict complications been tested. We hypothesize that a rules-based algorithm can successfully identify patients with the diseases in the Revised Cardiac Risk Index (RCRI). METHODS Patients undergoing surgery at the University of California, Los Angeles Health System between April 1, 2013 and July 1, 2016 and who had at least 2 previous office visits were included. For each disease in the RCRI except renal failure-congestive heart failure, ischemic heart disease, cerebrovascular disease, and diabetes mellitus-diagnosis algorithms were created based on diagnostic and standard clinical treatment criteria. For each disease state, the prevalence of the disease as determined by the algorithm, International Classification of Disease (ICD) code, and anesthesiologist's preoperative note were determined. Additionally, 400 American Society of Anesthesiologists classes III and IV cases were randomly chosen for manual review by an anesthesiologist. The sensitivity, specificity, accuracy, positive predictive value, negative predictive value, and area under the receiver operating characteristic curve were determined using the manual review as a gold standard. Last, the ability of the RCRI as calculated by each of the methods to predict in-hospital mortality was determined, and the time necessary to run the algorithms was calculated. RESULTS A total of 64,151 patients met inclusion criteria for the study. In general, the incidence of definite or likely disease determined by the algorithms was higher than that detected by the anesthesiologist. Additionally, in all disease states, the prevalence of disease was always lowest for the ICD codes, followed by the preoperative note, followed by the algorithms. In the subset of patients for whom the records were manually reviewed, the algorithms were generally the most sensitive and the ICD codes the most specific. When computing the modified RCRI using each of the methods, the modified RCRI from the algorithms predicted in-hospital mortality with an area under the receiver operating characteristic curve of 0.70 (0.67-0.73), which compared to 0.70 (0.67-0.72) for ICD codes and 0.64 (0.61-0.67) for the preoperative note. On average, the algorithms took 12.64 ± 1.20 minutes to run on 1.4 million patients. CONCLUSIONS Rules-based algorithms for disease in the RCRI can be created that perform with a similar discriminative ability as compared to physician notes and ICD codes but with significantly increased economies of scale.
Collapse
|
136
|
Lelong R, Soualmia LF, Grosjean J, Taalba M, Darmoni SJ. Building a Semantic Health Data Warehouse in the Context of Clinical Trials: Development and Usability Study. JMIR Med Inform 2019; 7:e13917. [PMID: 31859675 PMCID: PMC6942180 DOI: 10.2196/13917] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2019] [Revised: 08/02/2019] [Accepted: 08/19/2019] [Indexed: 01/08/2023] Open
Abstract
Background The huge amount of clinical, administrative, and demographic data recorded and maintained by hospitals can be consistently aggregated into health data warehouses with a uniform data model. In 2017, Rouen University Hospital (RUH) initiated the design of a semantic health data warehouse enabling both semantic description and retrieval of health information. Objective This study aimed to present a proof of concept of this semantic health data warehouse, based on the data of 250,000 patients from RUH, and to assess its ability to assist health professionals in prescreening eligible patients in a clinical trials context. Methods The semantic health data warehouse relies on 3 distinct semantic layers: (1) a terminology and ontology portal, (2) a semantic annotator, and (3) a semantic search engine and NoSQL (not only structured query language) layer to enhance data access performances. The system adopts an entity-centered vision that provides generic search capabilities able to express data requirements in terms of the whole set of interconnected conceptual entities that compose health information. Results We assessed the ability of the system to assist the search for 95 inclusion and exclusion criteria originating from 5 randomly chosen clinical trials from RUH. The system succeeded in fully automating 39% (29/74) of the criteria and was efficiently used as a prescreening tool for 73% (54/74) of them. Furthermore, the targeted sources of information and the search engine–related or data-related limitations that could explain the results for each criterion were also observed. Conclusions The entity-centered vision contrasts with the usual patient-centered vision adopted by existing systems. It enables more genericity in the information retrieval process. It also allows to fully exploit the semantic description of health information. Despite their semantic annotation, searching within clinical narratives remained the major challenge of the system. A finer annotation of the clinical texts and the addition of specific functionalities would significantly improve the results. The semantic aspect of the system combined with its generic entity-centered vision enables the processing of a large range of clinical questions. However, an important part of health information remains in clinical narratives, and we are currently investigating novel approaches (deep learning) to enhance the semantic annotation of those unstructured data.
Collapse
Affiliation(s)
- Romain Lelong
- Department of Biomedical Informatics, Rouen University Hospital, Rouen, France.,LITIS EA 4108, TIBS, Normandy University, Rouen, France
| | - Lina F Soualmia
- Department of Biomedical Informatics, Rouen University Hospital, Rouen, France.,LITIS EA 4108, TIBS, Normandy University, Rouen, France.,LIMICS U1142, Inserm, Sorbonne University, Paris, France
| | - Julien Grosjean
- Department of Biomedical Informatics, Rouen University Hospital, Rouen, France.,LIMICS U1142, Inserm, Sorbonne University, Paris, France
| | - Mehdi Taalba
- Department of Biomedical Informatics, Rouen University Hospital, Rouen, France
| | - Stéfan J Darmoni
- Department of Biomedical Informatics, Rouen University Hospital, Rouen, France.,LIMICS U1142, Inserm, Sorbonne University, Paris, France
| |
Collapse
|
137
|
Mercaldo ND, Brothers KB, Carrell DS, Clayton EW, Connolly JJ, Holm IA, Horowitz CR, Jarvik GP, Kitchner TE, Li R, McCarty CA, McCormick JB, McManus VD, Myers MF, Pankratz JJ, Shrubsole MJ, Smith ME, Stallings SC, Williams JL, Schildcrout JS. Enrichment sampling for a multi-site patient survey using electronic health records and census data. J Am Med Inform Assoc 2019; 26:219-227. [PMID: 30590688 DOI: 10.1093/jamia/ocy164] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2018] [Accepted: 11/15/2018] [Indexed: 11/14/2022] Open
Abstract
Objective We describe a stratified sampling design that combines electronic health records (EHRs) and United States Census (USC) data to construct the sampling frame and an algorithm to enrich the sample with individuals belonging to rarer strata. Materials and Methods This design was developed for a multi-site survey that sought to examine patient concerns about and barriers to participating in research studies, especially among under-studied populations (eg, minorities, low educational attainment). We defined sampling strata by cross-tabulating several socio-demographic variables obtained from EHR and augmented with census-block-level USC data. We oversampled rarer and historically underrepresented subpopulations. Results The sampling strategy, which included USC-supplemented EHR data, led to a far more diverse sample than would have been expected under random sampling (eg, 3-, 8-, 7-, and 12-fold increase in African Americans, Asians, Hispanics and those with less than a high school degree, respectively). We observed that our EHR data tended to misclassify minority races more often than majority races, and that non-majority races, Latino ethnicity, younger adult age, lower education, and urban/suburban living were each associated with lower response rates to the mailed surveys. Discussion We observed substantial enrichment from rarer subpopulations. The magnitude of the enrichment depends on the accuracy of the variables that define the sampling strata and the overall response rate. Conclusion EHR and USC data may be used to define sampling strata that in turn may be used to enrich the final study sample. This design may be of particular interest for studies of rarer and understudied populations.
Collapse
Affiliation(s)
- Nathaniel D Mercaldo
- Department of Radiology, Institute for Technology Assessment, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Kyle B Brothers
- Department of Pediatrics, University of Louisville, Louisville, Kentucky, USA
| | - David S Carrell
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
| | - Ellen W Clayton
- Center for Biomedical Ethics and Society, Vanderbilt University, Nashville, Tennessee, USA
| | - John J Connolly
- Center for Applied Genomics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Ingrid A Holm
- Division of Genetics and Genomics, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Carol R Horowitz
- Department of Population Health Science and Policy, Ichan School of Medicine at Mt. Sinai, New York, New York, USA
| | - Gail P Jarvik
- Department of Genome Sciences, University of Washington, Seattle, Washington, USA
| | - Terrie E Kitchner
- Center for Human Genetics, Marshfield Clinic Research Foundation, Marshfield, Wisconsin, USA
| | - Rongling Li
- Division of Genomic Medicine, National Human Genome Research Institute, Bethesda, Maryland, USA
| | - Catherine A McCarty
- Department of Family Medicine and Biobehavioral Health, University of Minnesota Medical School, Duluth, Minnesota, USA
| | | | - Valerie D McManus
- Biomedical Informatics Research Center, Marshfield Clinic Research Foundation, Marshfield, Wisconsin, USA
| | - Melanie F Myers
- Division of Human Genetics, Cincinnati Children's Hospital, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA
| | - Joshua J Pankratz
- Department of Information Technology, Mayo Clinic, Rochester, Minnesota, USA
| | - Martha J Shrubsole
- Vanderbilt Epidemiology Center, Vanderbilt University, Nashville, Tennessee, USA
| | - Maureen E Smith
- Center for Genetic Medicine, Northwestern University, Chicago, Illinois, USA
| | - Sarah C Stallings
- Division of Geriatric Medicine, Vanderbilt University, Nashville, Tennessee, USA
| | - Janet L Williams
- Genomic Medicine Institute, Geisinger, Danville, Pennsylvania, USA
| | | |
Collapse
|
138
|
Grundmeier RW, Xiao R, Ross RK, Ramos MJ, Karavite DJ, Michel JJ, Gerber JS, Coffin SE. Identifying surgical site infections in electronic health data using predictive models. J Am Med Inform Assoc 2019; 25:1160-1166. [PMID: 29982511 DOI: 10.1093/jamia/ocy075] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2018] [Accepted: 05/22/2018] [Indexed: 12/28/2022] Open
Abstract
Objective The objective was to prospectively derive and validate a prediction rule for detecting cases warranting investigation for surgical site infections (SSI) after ambulatory surgery. Methods We analysed electronic health record (EHR) data for children who underwent ambulatory surgery at one of 4 ambulatory surgical facilities. Using regularized logistic regression and random forests, we derived SSI prediction rules using 30 months of data (derivation set) and evaluated performance with data from the subsequent 10 months (validation set). Models were developed both with and without data extracted from free text. We also evaluated the presence of an antibiotic prescription within 60 days after surgery as an independent indicator of SSI evidence. Our goal was to exceed 80% sensitivity and 10% positive predictive value (PPV). Results We identified 234 surgeries with evidence of SSI among the 7910 surgeries available for analysis. We derived and validated an optimal prediction rule that included free text data using a random forest model (sensitivity = 0.9, PPV = 0.28). Presence of an antibiotic prescription had poor sensitivity (0.65) when applied to the derivation data but performed better when applied to the validation data (sensitivity = 0.84, PPV = 0.28). Conclusions EHR data can facilitate SSI surveillance with adequate sensitivity and PPV.
Collapse
Affiliation(s)
- Robert W Grundmeier
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA, USA.,Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA
| | - Rui Xiao
- Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA
| | - Rachael K Ross
- Center for Pediatric Clinical Effectiveness, Children's Hospital of Philadelphia Research Institute, Philadelphia, PA, USA
| | - Mark J Ramos
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA, USA
| | - Dean J Karavite
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA, USA
| | - Jeremy J Michel
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA, USA.,Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA
| | - Jeffrey S Gerber
- Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA.,Center for Pediatric Clinical Effectiveness, Children's Hospital of Philadelphia Research Institute, Philadelphia, PA, USA
| | - Susan E Coffin
- Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA.,Center for Pediatric Clinical Effectiveness, Children's Hospital of Philadelphia Research Institute, Philadelphia, PA, USA
| |
Collapse
|
139
|
Guo GN, Jonnagaddala J, Farshid S, Huser V, Reich C, Liaw ST. Comparison of the cohort selection performance of Australian Medicines Terminology to Anatomical Therapeutic Chemical mappings. J Am Med Inform Assoc 2019; 26:1237-1246. [PMID: 31545380 PMCID: PMC7647230 DOI: 10.1093/jamia/ocz143] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2019] [Revised: 06/10/2019] [Accepted: 07/22/2019] [Indexed: 12/16/2022] Open
Abstract
OBJECTIVE Electronic health records are increasingly utilized for observational and clinical research. Identification of cohorts using electronic health records is an important step in this process. Previous studies largely focused on the methods of cohort selection, but there is little evidence on the impact of underlying vocabularies and mappings between vocabularies used for cohort selection. We aim to compare the cohort selection performance using Australian Medicines Terminology to Anatomical Therapeutic Chemical (ATC) mappings from 2 different sources. These mappings were taken from the Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM) and the Pharmaceutical Benefits Scheme (PBS) schedule. MATERIALS AND METHODS We retrieved patients from the electronic Practice Based Research Network data repository using 3 ATC classification groups (A10, N02A, N06A). The retrieved patients were further verified manually and pooled to form a reference standard which was used to assess the accuracy of mappings using precision, recall, and F measure metrics. RESULTS The OMOP-CDM mappings identified 2.6%, 15.2%, and 24.4% more drugs than the PBS mappings in the A10, N02A and N06A groups respectively. Despite this, the PBS mappings generally performed the same in cohort selection as OMOP-CDM mappings except for the N02A Opioids group, where a significantly greater number of patients were retrieved. Both mappings exhibited variable recall, but perfect precision, with all drugs found to be correctly identified. CONCLUSION We found that 1 of the 3 ATC groups had a significant difference and this affected cohort selection performance. Our findings highlighted that underlying terminology mappings can greatly impact cohort selection accuracy. Clinical researchers should carefully evaluate vocabulary mapping sources including methodologies used to develop those mappings.
Collapse
Affiliation(s)
- Guan N Guo
- School of Public Health and Community Medicine, University of New South Wales, Sydney, Australia
- WHO Collaborating Centre for eHealth, University of New South Wales, Sydney, Australia
| | - Jitendra Jonnagaddala
- School of Public Health and Community Medicine, University of New South Wales, Sydney, Australia
- WHO Collaborating Centre for eHealth, University of New South Wales, Sydney, Australia
| | - Sanjay Farshid
- School of Public Health and Community Medicine, University of New South Wales, Sydney, Australia
| | - Vojtech Huser
- Lister Hill National Centre for Biomedical Communications, National Library of Medicine National Institutes of Health, Bethesda, Maryland, USA
| | | | - Siaw-Teng Liaw
- School of Public Health and Community Medicine, University of New South Wales, Sydney, Australia
- WHO Collaborating Centre for eHealth, University of New South Wales, Sydney, Australia
| |
Collapse
|
140
|
Chartash D, Paek H, Dziura JD, Ross BK, Nogee DP, Boccio E, Hines C, Schott AM, Jeffery MM, Patel MD, Platts-Mills TF, Ahmed O, Brandt C, Couturier K, Melnick E. Identifying Opioid Use Disorder in the Emergency Department: Multi-System Electronic Health Record-Based Computable Phenotype Derivation and Validation Study. JMIR Med Inform 2019; 7:e15794. [PMID: 31674913 PMCID: PMC6913746 DOI: 10.2196/15794] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2019] [Revised: 09/27/2019] [Accepted: 10/01/2019] [Indexed: 01/10/2023] Open
Abstract
BACKGROUND Deploying accurate computable phenotypes in pragmatic trials requires a trade-off between precise and clinically sensical variable selection. In particular, evaluating the medical encounter to assess a pattern leading to clinically significant impairment or distress indicative of disease is a difficult modeling challenge for the emergency department. OBJECTIVE This study aimed to derive and validate an electronic health record-based computable phenotype to identify emergency department patients with opioid use disorder using physician chart review as a reference standard. METHODS A two-algorithm computable phenotype was developed and evaluated using structured clinical data across 13 emergency departments in two large health care systems. Algorithm 1 combined clinician and billing codes. Algorithm 2 used chief complaint structured data suggestive of opioid use disorder. To evaluate the algorithms in both internal and external validation phases, two emergency medicine physicians, with a third acting as adjudicator, reviewed a pragmatic sample of 231 charts: 125 internal validation (75 positive and 50 negative), 106 external validation (56 positive and 50 negative). RESULTS Cohen kappa, measuring agreement between reviewers, for the internal and external validation cohorts was 0.95 and 0.93, respectively. In the internal validation phase, Algorithm 1 had a positive predictive value (PPV) of 0.96 (95% CI 0.863-0.995) and a negative predictive value (NPV) of 0.98 (95% CI 0.893-0.999), and Algorithm 2 had a PPV of 0.8 (95% CI 0.593-0.932) and an NPV of 1.0 (one-sided 97.5% CI 0.863-1). In the external validation phase, the phenotype had a PPV of 0.95 (95% CI 0.851-0.989) and an NPV of 0.92 (95% CI 0.807-0.978). CONCLUSIONS This phenotype detected emergency department patients with opioid use disorder with high predictive values and reliability. Its algorithms were transportable across health care systems and have potential value for both clinical and research purposes.
Collapse
Affiliation(s)
- David Chartash
- Yale Center for Medical Informatics, Yale University School of Medicine, New Haven, CT, United States
| | - Hyung Paek
- Information Technology Services, Yale New Haven Health, New Haven, CT, United States
| | - James D Dziura
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Bill K Ross
- North Carolina Translational and Clinical Sciences Institute, University of North Carolina School of Medicine, Chapel Hill, NC, United States
| | - Daniel P Nogee
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Eric Boccio
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Cory Hines
- Department of Emergency Medicine, University of North Carolina School of Medicine, Chapel Hill, NC, United States
| | - Aaron M Schott
- Department of Emergency Medicine, University of North Carolina School of Medicine, Chapel Hill, NC, United States
| | - Molly M Jeffery
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Mehul D Patel
- Department of Emergency Medicine, University of North Carolina School of Medicine, Chapel Hill, NC, United States
| | - Timothy F Platts-Mills
- Department of Emergency Medicine, University of North Carolina School of Medicine, Chapel Hill, NC, United States
| | - Osama Ahmed
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Cynthia Brandt
- Yale Center for Medical Informatics, Yale University School of Medicine, New Haven, CT, United States
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Katherine Couturier
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Edward Melnick
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| |
Collapse
|
141
|
Spasic I, Krzeminski D, Corcoran P, Balinsky A. Cohort Selection for Clinical Trials From Longitudinal Patient Records: Text Mining Approach. JMIR Med Inform 2019; 7:e15980. [PMID: 31674914 PMCID: PMC6913747 DOI: 10.2196/15980] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2019] [Revised: 09/29/2019] [Accepted: 10/02/2019] [Indexed: 12/17/2022] Open
Abstract
Background Clinical trials are an important step in introducing new interventions into clinical practice by generating data on their safety and efficacy. Clinical trials need to ensure that participants are similar so that the findings can be attributed to the interventions studied and not to some other factors. Therefore, each clinical trial defines eligibility criteria, which describe characteristics that must be shared by the participants. Unfortunately, the complexities of eligibility criteria may not allow them to be translated directly into readily executable database queries. Instead, they may require careful analysis of the narrative sections of medical records. Manual screening of medical records is time consuming, thus negatively affecting the timeliness of the recruitment process. Objective Track 1 of the 2018 National Natural Language Processing Clinical Challenge focused on the task of cohort selection for clinical trials, aiming to answer the following question: Can natural language processing be applied to narrative medical records to identify patients who meet eligibility criteria for clinical trials? The task required the participating systems to analyze longitudinal patient records to determine if the corresponding patients met the given eligibility criteria. We aimed to describe a system developed to address this task. Methods Our system consisted of 13 classifiers, one for each eligibility criterion. All classifiers used a bag-of-words document representation model. To prevent the loss of relevant contextual information associated with such representation, a pattern-matching approach was used to extract context-sensitive features. They were embedded back into the text as lexically distinguishable tokens, which were consequently featured in the bag-of-words representation. Supervised machine learning was chosen wherever a sufficient number of both positive and negative instances was available to learn from. A rule-based approach focusing on a small set of relevant features was chosen for the remaining criteria. Results The system was evaluated using microaveraged F measure. Overall, 4 machine algorithms, including support vector machine, logistic regression, naïve Bayesian classifier, and gradient tree boosting (GTB), were evaluated on the training data using 10–fold cross-validation. Overall, GTB demonstrated the most consistent performance. Its performance peaked when oversampling was used to balance the training data. The final evaluation was performed on previously unseen test data. On average, the F measure of 89.04% was comparable to 3 of the top ranked performances in the shared task (91.11%, 90.28%, and 90.21%). With an F measure of 88.14%, we significantly outperformed these systems (81.03%, 78.50%, and 70.81%) in identifying patients with advanced coronary artery disease. Conclusions The holdout evaluation provides evidence that our system was able to identify eligible patients for the given clinical trial with high accuracy. Our approach demonstrates how rule-based knowledge infusion can improve the performance of machine learning algorithms even when trained on a relatively small dataset.
Collapse
Affiliation(s)
- Irena Spasic
- School of Computer Science & Informatics, Cardiff University, Cardiff, United Kingdom
| | | | - Padraig Corcoran
- School of Computer Science & Informatics, Cardiff University, Cardiff, United Kingdom
| | | |
Collapse
|
142
|
Abstract
Electronic Health Records (EHR) are a rich repository of valuable clinical information that exist in primary and secondary care databases. In order to utilize EHRs for medical observational research a range of algorithms for automatically identifying individuals with a specific phenotype have been developed. This review summarizes and offers a critical evaluation of the literature relating to studies conducted into the development of EHR phenotyping systems. This review describes phenotyping systems and techniques based on structured and unstructured EHR data. Articles published on PubMed and Google scholar between 2013 and 2017 have been reviewed, using search terms derived from Medical Subject Headings (MeSH). The popularity of using Natural Language Processing (NLP) techniques in extracting features from narrative text has increased. This increased attention is due to the availability of open source NLP algorithms, combined with accuracy improvement. In this review, Concept extraction is the most popular NLP technique since it has been used by more than 50% of the reviewed papers to extract features from EHR. High-throughput phenotyping systems using unsupervised machine learning techniques have gained more popularity due to their ability to efficiently and automatically extract a phenotype with minimal human effort.
Collapse
|
143
|
Newman-Griffis D, Porcino J, Zirikly A, Thieu T, Camacho Maldonado J, Ho PS, Ding M, Chan L, Rasch E. Broadening horizons: the case for capturing function and the role of health informatics in its use. BMC Public Health 2019; 19:1288. [PMID: 31615472 PMCID: PMC6794808 DOI: 10.1186/s12889-019-7630-3] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2018] [Accepted: 09/16/2019] [Indexed: 12/18/2022] Open
Abstract
Background Human activity and the interaction between health conditions and activity is a critical part of understanding the overall function of individuals. The World Health Organization’s International Classification of Functioning, Disability and Health (ICF) models function as all aspects of an individual’s interaction with the world, including organismal concepts such as individual body structures, functions, and pathologies, as well as the outcomes of the individual’s interaction with their environment, referred to as activity and participation. Function, particularly activity and participation outcomes, is an important indicator of health at both the level of an individual and the population level, as it is highly correlated with quality of life and a critical component of identifying resource needs. Since it reflects the cumulative impact of health conditions on individuals and is not disease specific, its use as a health indicator helps to address major barriers to holistic, patient-centered care that result from multiple, and often competing, disease specific interventions. While the need for better information on function has been widely endorsed, this has not translated into its routine incorporation into modern health systems. Purpose We present the importance of capturing information on activity as a core component of modern health systems and identify specific steps and analytic methods that can be used to make it more available to utilize in improving patient care. We identify challenges in the use of activity and participation information, such as a lack of consistent documentation and diversity of data specificity and representation across providers, health systems, and national surveys. We describe how activity and participation information can be more effectively captured, and how health informatics methodologies, including natural language processing (NLP), can enable automatically locating, extracting, and organizing this information on a large scale, supporting standardization and utilization with minimal additional provider burden. We examine the analytic requirements and potential challenges of capturing this information with informatics, and describe how data-driven techniques can combine with common standards and documentation practices to make activity and participation information standardized and accessible for improving patient care. Recommendations We recommend four specific actions to improve the capture and analysis of activity and participation information throughout the continuum of care: (1) make activity and participation annotation standards and datasets available to the broader research community; (2) define common research problems in automatically processing activity and participation information; (3) develop robust, machine-readable ontologies for function that describe the components of activity and participation information and their relationships; and (4) establish standards for how and when to document activity and participation status during clinical encounters. We further provide specific short-term goals to make significant progress in each of these areas within a reasonable time frame.
Collapse
Affiliation(s)
- Denis Newman-Griffis
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA. .,Department of Computer Science and Engineering, The Ohio State University, 2015 Neil Avenue, DL 395, Columbus, OH, 43210, USA.
| | - Julia Porcino
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA
| | - Ayah Zirikly
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA
| | - Thanh Thieu
- Department of Computer Science, Oklahoma State University, 116-A MSCS, Stillwater, OK, 74078, USA
| | - Jonathan Camacho Maldonado
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA
| | - Pei-Shu Ho
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA
| | - Min Ding
- Information Technology Laboratory, National Institute of Standards and Technology, 100 Bureau Drive, Gaithersburg, MD, 20899, USA
| | - Leighton Chan
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA
| | - Elizabeth Rasch
- Rehabilitation Medicine Department, National Institutes of Health, Mark O. Hatfield Clinical Research Center, 6707 Democracy Boulevard, Suite 856, MSC 5493, Bethesda, MD, 20892, USA
| |
Collapse
|
144
|
Liu D, Baskett W, Beversdorf D, Shyu CR. Exploratory Data Mining for Subgroup Cohort Discoveries and Prioritization. IEEE J Biomed Health Inform 2019; 24:1456-1468. [PMID: 31494566 PMCID: PMC9341221 DOI: 10.1109/jbhi.2019.2939149] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Finding small homogeneous subgroup cohorts in large heterogeneous populations is a critical process for hypothesis development in biomedical research. Concurrent computational approaches are still lacking in robust answers to the question "what hypotheses are likely to be novel and to produce clinically relevant results with well thought-out study designs?" We have developed a novel subgroup discovery method which employs a deep exploratory mining process to slice and dice thousands of potential subpopulations and prioritize potential cohorts based on their explainable contrast patterns and which may provide interventionable insights. We conducted computational experiments on both synthesized data and a clinical autism data set to assess performance quantitatively for coverage of pre-defined cohorts and qualitatively for novel knowledge discovery, respectively. We also conducted a scaling analysis using a distributed computing environment to suggest computational resource needs for when the subpopulation number increases. This work will provide a robust data-driven framework to automatically tailor potential interventions for precision health.
Collapse
|
145
|
Obeid JS, Heider PM, Weeda ER, Matuskowitz AJ, Carr CM, Gagnon K, Crawford T, Meystre SM. Impact of De-Identification on Clinical Text Classification Using Traditional and Deep Learning Classifiers. Stud Health Technol Inform 2019; 264:283-287. [PMID: 31437930 PMCID: PMC6779034 DOI: 10.3233/shti190228] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Clinical text de-identification enables collaborative research while protecting patient privacy and confidentiality; however, concerns persist about the reduction in the utility of the de-identified text for information extraction and machine learning tasks. In the context of a deep learning experiment to detect altered mental status in emergency department provider notes, we tested several classifiers on clinical notes in their original form and on their automatically de-identified counterpart. We tested both traditional bag-of-words based machine learning models as well as word-embedding based deep learning models. We evaluated the models on 1,113 history of present illness notes. A total of 1,795 protected health information tokens were replaced in the de-identification process across all notes. The deep learning models had the best performance with accuracies of 95% on both original and de-identified notes. However, there was no significant difference in the performance of any of the models on the original vs. the de-identified notes.
Collapse
Affiliation(s)
- Jihad S. Obeid
- Biomedical Informatics Center, Medical University of South
Carolina, Charleston, SC, USA
| | - Paul M. Heider
- Biomedical Informatics Center, Medical University of South
Carolina, Charleston, SC, USA
| | - Erin R. Weeda
- Department of Clinical Pharmacy and Outcome Sciences,
Medical University of South Carolina, Charleston, SC, USA
| | - Andrew J. Matuskowitz
- Department of Emergency Medicine, Medical University of
South Carolina, Charleston, SC, USA
| | - Christine M. Carr
- Biomedical Informatics Center, Medical University of South
Carolina, Charleston, SC, USA
- Department of Emergency Medicine, Medical University of
South Carolina, Charleston, SC, USA
| | - Kevin Gagnon
- Department of Computer Science, University of South
Carolina, Columbia, SC, USA
| | - Tami Crawford
- Biomedical Informatics Center, Medical University of South
Carolina, Charleston, SC, USA
| | - Stephane M. Meystre
- Biomedical Informatics Center, Medical University of South
Carolina, Charleston, SC, USA
| |
Collapse
|
146
|
Zeng Z, Zhao Y, Sun M, Vo AH, Starren J, Luo Y. Rich Text Formatted EHR Narratives: A Hidden and Ignored Trove. Stud Health Technol Inform 2019; 264:472-476. [PMID: 31437968 PMCID: PMC8060951 DOI: 10.3233/shti190266] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
This study presents an approach for mining structured information from clinical narratives in Electronic Health Records (EHRs) by using Rich Text Formatted (RTF) records. RTF is adopted by many medical information management systems. There is rich structural information in these files which can be extracted and interpreted, yet such information is largely ignored. We investigate multiple types of EHR narratives in the Enterprise Data Warehouse from a multisite large healthcare chain consisting of both, an academic medical center and community hospitals. We focus on the RTF constructs related to tables and sections that are not available in plain text EHR narratives. We show how to parse these RTF constructs, analyze their prevalence and characteristics in the context of multiple types of EHR narratives. Our case study demonstrates the additional utility of the features derived from RTF constructs over plain text oriented NLP.
Collapse
Affiliation(s)
- Zexian Zeng
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Yuan Zhao
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Mengxin Sun
- Hospital Medicine, Northwestern Memorial Hospital, Chicago, IL, USA
| | - Andy H Vo
- Committee on Developmental Biology and Regenerative Medicine, The University of Chicago, Chicago, IL, USA
| | - Justin Starren
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Yuan Luo
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| |
Collapse
|
147
|
Obeid JS, Weeda ER, Matuskowitz AJ, Gagnon K, Crawford T, Carr CM, Frey LJ. Automated detection of altered mental status in emergency department clinical notes: a deep learning approach. BMC Med Inform Decis Mak 2019; 19:164. [PMID: 31426779 PMCID: PMC6701023 DOI: 10.1186/s12911-019-0894-9] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2019] [Accepted: 08/11/2019] [Indexed: 12/18/2022] Open
Abstract
BACKGROUND Machine learning has been used extensively in clinical text classification tasks. Deep learning approaches using word embeddings have been recently gaining momentum in biomedical applications. In an effort to automate the identification of altered mental status (AMS) in emergency department provider notes for the purpose of decision support, we compare the performance of classic bag-of-words-based machine learning classifiers and novel deep learning approaches. METHODS We used a case-control study design to extract an adequate number of clinical notes with AMS and non-AMS based on ICD codes. The notes were parsed to extract the history of present illness, which was used as the clinical text for the classifiers. The notes were manually labeled by clinicians. As a baseline for comparison, we tested several traditional bag-of-words based classifiers. We then tested several deep learning models using a convolutional neural network architecture with three different types of word embeddings, a pre-trained word2vec model and two models without pre-training but with different word embedding dimensions. RESULTS We evaluated the models on 1130 labeled notes from the emergency department. The deep learning models had the best overall performance with an area under the ROC curve of 98.5% and an accuracy of 94.5%. Pre-training word embeddings on the unlabeled corpus reduced training iterations and had performance that was statistically no different than the other deep learning models. CONCLUSION This supervised deep learning approach performs exceedingly well for the detection of AMS symptoms in clinical text in our environment. Further work is needed for the generalizability of these findings, including evaluation of these models in other types of clinical notes and other environments. The results seem promising for the ultimate use of these types of classifiers in combination with other information derived from the electronic health records as input for clinical decision support.
Collapse
Affiliation(s)
- Jihad S Obeid
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, USA.
- Department of Public Health Sciences, Medical University of South Carolina, Charleston, SC, USA.
| | - Erin R Weeda
- Department of Clinical Pharmacy and Outcome Sciences, Medical University of South Carolina, Charleston, SC, USA
| | - Andrew J Matuskowitz
- Department of Emergency Medicine, Medical University of South Carolina, Charleston, SC, USA
| | - Kevin Gagnon
- Department of Computer Science and Engineering, University of South Carolina, Columbia, SC, USA
| | - Tami Crawford
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, USA
| | - Christine M Carr
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, USA
- Department of Emergency Medicine, Medical University of South Carolina, Charleston, SC, USA
| | - Lewis J Frey
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, USA
- Department of Public Health Sciences, Medical University of South Carolina, Charleston, SC, USA
| |
Collapse
|
148
|
Machine Learning for Feature Selection and Cluster Analysis in Drug Utilisation Research. CURR EPIDEMIOL REP 2019. [DOI: 10.1007/s40471-019-00211-7] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
149
|
Patterson BW, Jacobsohn GC, Shah MN, Song Y, Maru A, Venkatesh AK, Zhong M, Taylor K, Hamedani AG, Mendonça EA. Development and validation of a pragmatic natural language processing approach to identifying falls in older adults in the emergency department. BMC Med Inform Decis Mak 2019; 19:138. [PMID: 31331322 PMCID: PMC6647058 DOI: 10.1186/s12911-019-0843-7] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2019] [Accepted: 06/20/2019] [Indexed: 01/30/2023] Open
Abstract
BACKGROUND Falls among older adults are both a common reason for presentation to the emergency department, and a major source of morbidity and mortality. It is critical to identify fall patients quickly and reliably during, and immediately after, emergency department encounters in order to deliver appropriate care and referrals. Unfortunately, falls are difficult to identify without manual chart review, a time intensive process infeasible for many applications including surveillance and quality reporting. Here we describe a pragmatic NLP approach to automating fall identification. METHODS In this single center retrospective review, 500 emergency department provider notes from older adult patients (age 65 and older) were randomly selected for analysis. A simple, rules-based NLP algorithm for fall identification was developed and evaluated on a development set of 1084 notes, then compared with identification by consensus of trained abstractors blinded to NLP results. RESULTS The NLP pipeline demonstrated a recall (sensitivity) of 95.8%, specificity of 97.4%, precision of 92.0%, and F1 score of 0.939 for identifying fall events within emergency physician visit notes, as compared to gold standard manual abstraction by human coders. CONCLUSIONS Our pragmatic NLP algorithm was able to identify falls in ED notes with excellent precision and recall, comparable to that of more labor-intensive manual abstraction. This finding offers promise not just for improving research methods, but as a potential for identifying patients for targeted interventions, quality measure development and epidemiologic surveillance.
Collapse
Affiliation(s)
- Brian W Patterson
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA. .,Health Innovation Program, University of Wisconsin-Madison, Madison, WI, 53705, USA.
| | - Gwen C Jacobsohn
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
| | - Manish N Shah
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.,Department of Medicine, Division of Geriatrics, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States
| | - Yiqiang Song
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, USA.,Department of Pediatrics and Department of Biostatistics, Indiana University School of Medicine, Indianapolis, IN, USA
| | - Apoorva Maru
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
| | - Arjun K Venkatesh
- Department of Emergency Medicine and Center for Outcomes Research and Evaluation, Yale University School of Medicine, New Haven, CT, USA
| | - Monica Zhong
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
| | - Katherine Taylor
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
| | - Azita G Hamedani
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
| | - Eneida A Mendonça
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, USA.,Department of Pediatrics and Department of Biostatistics, Indiana University School of Medicine, Indianapolis, IN, USA.,Regenstrief Institute, Indianapolis, IN, USA
| |
Collapse
|
150
|
Cheung KS, Leung WK, Seto WK. Application of Big Data analysis in gastrointestinal research. World J Gastroenterol 2019; 25:2990-3008. [PMID: 31293336 PMCID: PMC6603810 DOI: 10.3748/wjg.v25.i24.2990] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/11/2019] [Revised: 04/14/2019] [Accepted: 04/29/2019] [Indexed: 02/06/2023] Open
Abstract
Big Data, which are characterized by certain unique traits like volume, velocity and value, have revolutionized the research of multiple fields including medicine. Big Data in health care are defined as large datasets that are collected routinely or automatically, and stored electronically. With the rapidly expanding volume of health data collection, it is envisioned that the Big Data approach can improve not only individual health, but also the performance of health care systems. The application of Big Data analysis in the field of gastroenterology and hepatology research has also opened new research approaches. While it retains most of the advantages and avoids some of the disadvantages of traditional observational studies (case-control and prospective cohort studies), it allows for phenomapping of disease heterogeneity, enhancement of drug safety, as well as development of precision medicine, prediction models and personalized treatment. Unlike randomized controlled trials, it reflects the real-world situation and studies patients who are often under-represented in randomized controlled trials. However, residual and/or unmeasured confounding remains a major concern, which requires meticulous study design and various statistical adjustment methods. Other potential drawbacks include data validity, missing data, incomplete data capture due to the unavailability of diagnosis codes for certain clinical situations, and individual privacy. With continuous technological advances, some of the current limitations with Big Data may be further minimized. This review will illustrate the use of Big Data research on gastrointestinal and liver diseases using recently published examples.
Collapse
Affiliation(s)
- Ka-Shing Cheung
- Department of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong, China
- Department of Medicine, The University of Hong Kong-Shenzhen Hospital, Shenzhen 518053, Guangdong Province, China
| | - Wai K Leung
- Department of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong, China
| | - Wai-Kay Seto
- Department of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong, China
- Department of Medicine, The University of Hong Kong-Shenzhen Hospital, Shenzhen 518053, Guangdong Province, China
| |
Collapse
|