1
|
Bejan CA, Ripperger M, Wilimitis D, Ahmed R, Kang J, Robinson K, Morley TJ, Ruderfer DM, Walsh CG. Improving ascertainment of suicidal ideation and suicide attempt with natural language processing. Sci Rep 2022; 12:15146. [PMID: 36071081 PMCID: PMC9452591 DOI: 10.1038/s41598-022-19358-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2022] [Accepted: 08/29/2022] [Indexed: 12/03/2022] Open
Abstract
Methods relying on diagnostic codes to identify suicidal ideation and suicide attempt in Electronic Health Records (EHRs) at scale are suboptimal because suicide-related outcomes are heavily under-coded. We propose to improve the ascertainment of suicidal outcomes using natural language processing (NLP). We developed information retrieval methodologies to search over 200 million notes from the Vanderbilt EHR. Suicide query terms were extracted using word2vec. A weakly supervised approach was designed to label cases of suicidal outcomes. The NLP validation of the top 200 retrieved patients showed high performance for suicidal ideation (area under the receiver operator curve [AUROC]: 98.6, 95% confidence interval [CI] 97.1-99.5) and suicide attempt (AUROC: 97.3, 95% CI 95.2-98.7). Case extraction produced the best performance when combining NLP and diagnostic codes and when accounting for negated suicide expressions in notes. Overall, we demonstrated that scalable and accurate NLP methods can be developed to identify suicidal behavior in EHRs to enhance prevention efforts, predictive models, and precision medicine.
Collapse
Affiliation(s)
- Cosmin A Bejan
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Vanderbilt University School of Medicine, 2525 West End Avenue, Suite 1500, Nashville, TN, 37232, USA.
| | - Michael Ripperger
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Vanderbilt University School of Medicine, 2525 West End Avenue, Suite 1500, Nashville, TN, 37232, USA
| | - Drew Wilimitis
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Vanderbilt University School of Medicine, 2525 West End Avenue, Suite 1500, Nashville, TN, 37232, USA
| | - Ryan Ahmed
- Department of Medicine, Vanderbilt University Medical Center, Nashville, USA
| | - JooEun Kang
- Division of Genetic Medicine, Department of Medicine, Vanderbilt Genetics Institute, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Katelyn Robinson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Vanderbilt University School of Medicine, 2525 West End Avenue, Suite 1500, Nashville, TN, 37232, USA
| | - Theodore J Morley
- Division of Genetic Medicine, Department of Medicine, Vanderbilt Genetics Institute, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Douglas M Ruderfer
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Vanderbilt University School of Medicine, 2525 West End Avenue, Suite 1500, Nashville, TN, 37232, USA
- Division of Genetic Medicine, Department of Medicine, Vanderbilt Genetics Institute, Vanderbilt University Medical Center, Nashville, TN, USA
- Department of Psychiatry and Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Colin G Walsh
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Vanderbilt University School of Medicine, 2525 West End Avenue, Suite 1500, Nashville, TN, 37232, USA
- Department of Medicine, Vanderbilt University Medical Center, Nashville, USA
- Department of Psychiatry and Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| |
Collapse
|
2
|
Jujjavarapu C, Pejaver V, Cohen TA, Mooney SD, Heagerty PJ, Jarvik JG. A Comparison of Natural Language Processing Methods for the Classification of Lumbar Spine Imaging Findings Related to Lower Back Pain. Acad Radiol 2022; 29 Suppl 3:S188-S200. [PMID: 34862122 PMCID: PMC8917985 DOI: 10.1016/j.acra.2021.09.005] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2021] [Revised: 08/22/2021] [Accepted: 09/04/2021] [Indexed: 11/28/2022]
Abstract
RATIONALE AND OBJECTIVES The use of natural language processing (NLP) in radiology provides an opportunity to assist clinicians with phenotyping patients. However, the performance and generalizability of NLP across healthcare systems is uncertain. We assessed the performance within and generalizability across four healthcare systems of different NLP representational methods, coupled with elastic-net logistic regression to classify lower back pain-related findings from lumbar spine imaging reports. MATERIALS AND METHODS We used a dataset of 871 X-ray and magnetic resonance imaging reports sampled from a prospective study across four healthcare systems between October 2013 and September 2016. We annotated each report for 26 findings potentially related to lower back pain. Our framework applied four different NLP methods to convert text into feature sets (representations). For each representation, our framework used an elastic-net logistic regression model for each finding (i.e., 26 binary or "one-vs.-rest" classification models). For performance evaluation, we split data into training (80%, 697/871) and testing (20%, 174/871). In the training set, we used cross validation to identify the optimal hyperparameter value and then retrained on the full training set. We then assessed performance based on area under the curve (AUC) for the test set. We repeated this process 25 times with each repeat using a different random train/test split of the data, so that we could estimate 95% confidence intervals, and assess significant difference in performance between representations. For generalizability evaluation, we trained models on data from three healthcare systems with cross validation and then tested on the fourth. We repeated this process for each system, then calculated mean and standard deviation (SD) of AUC across the systems. RESULTS For individual representations, n-grams had the best average performance across all 26 findings (AUC: 0.960). For generalizability, document embeddings had the most consistent average performance across systems (SD: 0.010). Out of these 26 findings, we considered eight as potentially clinically important (any stenosis, central stenosis, lateral stenosis, foraminal stenosis, disc extrusion, nerve root displacement compression, endplate edema, and listhesis grade 2) since they have a relatively greater association with a history of lower back pain compared to the remaining 18 classes. We found a similar pattern for these eight in which n-grams and document embeddings had the best average performance (AUC: 0.954) and generalizability (SD: 0.007), respectively. CONCLUSION Based on performance assessment, we found that n-grams is the preferred method if classifier development and deployment occur at the same system. However, for deployment at multiple systems outside of the development system, or potentially if physician behavior changes within a system, one should consider document embeddings since embeddings appear to have the most consistent performance across systems.
Collapse
Affiliation(s)
- Chethan Jujjavarapu
- Department of Biomedical Informatics and Medical Education, School
of Medicine, University of Washington, Seattle, Washington
| | - Vikas Pejaver
- Department of Biomedical Informatics and Medical Education, School
of Medicine, University of Washington, Seattle, Washington
| | - Trevor A. Cohen
- Department of Biomedical Informatics and Medical Education, School
of Medicine, University of Washington, Seattle, Washington
| | - Sean D. Mooney
- Department of Biomedical Informatics and Medical Education, School
of Medicine, University of Washington, Seattle, Washington
| | - Patrick J. Heagerty
- Department of Biostatistics, University of Washington, Seattle,
Washington,Center for Biomedical Statistics, University of Washington,
Seattle, Washington
| | - Jeffrey G. Jarvik
- Department of Radiology, University of Washington, 1959 NE Pacific
Street, Seattle WA 98195,Department of Neurological Surgery, University of Washington,
Seattle, Washington,Department of Health Services, University of Washington, Seattle
Washington,Clinical Learning, Evidence And Research Center, University of
Washington, Seattle, Washington
| |
Collapse
|
3
|
Lybarger K, Ostendorf M, Thompson M, Yetisgen M. Extracting COVID-19 diagnoses and symptoms from clinical text: A new annotated corpus and neural event extraction framework. J Biomed Inform 2021; 117:103761. [PMID: 33781918 PMCID: PMC7997694 DOI: 10.1016/j.jbi.2021.103761] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2020] [Revised: 03/02/2021] [Accepted: 03/20/2021] [Indexed: 12/29/2022]
Abstract
Coronavirus disease 2019 (COVID-19) is a global pandemic. Although much has been learned about the novel coronavirus since its emergence, there are many open questions related to tracking its spread, describing symptomology, predicting the severity of infection, and forecasting healthcare utilization. Free-text clinical notes contain critical information for resolving these questions. Data-driven, automatic information extraction models are needed to use this text-encoded information in large-scale studies. This work presents a new clinical corpus, referred to as the COVID-19 Annotated Clinical Text (CACT) Corpus, which comprises 1,472 notes with detailed annotations characterizing COVID-19 diagnoses, testing, and clinical presentation. We introduce a span-based event extraction model that jointly extracts all annotated phenomena, achieving high performance in identifying COVID-19 and symptom events with associated assertion values (0.83-0.97 F1 for events and 0.73-0.79 F1 for assertions). Our span-based event extraction model outperforms an extractor built on MetaMapLite for the identification of symptoms with assertion values. In a secondary use application, we predicted COVID-19 test results using structured patient data (e.g. vital signs and laboratory results) and automatically extracted symptom information, to explore the clinical presentation of COVID-19. Automatically extracted symptoms improve COVID-19 prediction performance, beyond structured data alone.
Collapse
Affiliation(s)
- Kevin Lybarger
- Biomedical & Health Informatics, University of Washington, Box 358047, Seattle, WA 98109, USA.
| | - Mari Ostendorf
- Department of Electrical & Computer Engineering, University of Washington, Campus Box 352500 185, Seattle, WA 98195-2500, USA
| | - Matthew Thompson
- Department of Family Medicine, University of Washington, Box 354696, Seattle, WA 98195-2500, USA
| | - Meliha Yetisgen
- Biomedical & Health Informatics, University of Washington, Box 358047, Seattle, WA 98109, USA
| |
Collapse
|
4
|
Kim Y, Heider PM, Lally IR, Meystre SM. A Hybrid Model for Family History Information Identification and Relation Extraction: Development and Evaluation of an End-to-End Information Extraction System. JMIR Med Inform 2021; 9:e22797. [PMID: 33885370 PMCID: PMC8103307 DOI: 10.2196/22797] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2020] [Revised: 12/15/2020] [Accepted: 02/19/2021] [Indexed: 01/02/2023] Open
Abstract
BACKGROUND Family history information is important to assess the risk of inherited medical conditions. Natural language processing has the potential to extract this information from unstructured free-text notes to improve patient care and decision making. We describe the end-to-end information extraction system the Medical University of South Carolina team developed when participating in the 2019 National Natural Language Processing Clinical Challenge (n2c2)/Open Health Natural Language Processing (OHNLP) shared task. OBJECTIVE This task involves identifying mentions of family members and observations in electronic health record text notes and recognizing the 2 types of relations (family member-living status relations and family member-observation relations). Our system aims to achieve a high level of performance by integrating heuristics and advanced information extraction methods. Our efforts also include improving the performance of 2 subtasks by exploiting additional labeled data and clinical text-based embedding models. METHODS We present a hybrid method that combines machine learning and rule-based approaches. We implemented an end-to-end system with multiple information extraction and attribute classification components. For entity identification, we trained bidirectional long short-term memory deep learning models. These models incorporated static word embeddings and context-dependent embeddings. We created a voting ensemble that combined the predictions of all individual models. For relation extraction, we trained 2 relation extraction models. The first model determined the living status of each family member. The second model identified observations associated with each family member. We implemented online gradient descent models to extract related entity pairs. As part of postchallenge efforts, we used the BioCreative/OHNLP 2018 corpus and trained new models with the union of these 2 datasets. We also pretrained language models using clinical notes from the Medical Information Mart for Intensive Care (MIMIC-III) clinical database. RESULTS The voting ensemble achieved better performance than individual classifiers. In the entity identification task, our top-performing system reached a precision of 78.90% and a recall of 83.84%. Our natural language processing system for entity identification took 3rd place out of 17 teams in the challenge. We ranked 4th out of 9 teams in the relation extraction task. Our system substantially benefited from the combination of the 2 datasets. Compared to our official submission with F1 scores of 81.30% and 64.94% for entity identification and relation extraction, respectively, the revised system yielded significantly better performance (P<.05) with F1 scores of 86.02% and 72.48%, respectively. CONCLUSIONS We demonstrated that a hybrid model could be used to successfully extract family history information recorded in unstructured free-text notes. In this study, our approach to entity identification as a sequence labeling problem produced satisfactory results. Our postchallenge efforts significantly improved performance by leveraging additional labeled data and using word vector representations learned from large collections of clinical notes.
Collapse
Affiliation(s)
- Youngjun Kim
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, United States
| | - Paul M Heider
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, United States
| | - Isabel Rh Lally
- Department of Computer Science, College of Charleston, Charleston, SC, United States
| | - Stéphane M Meystre
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, United States
| |
Collapse
|
5
|
Bejan CA, Cahill KN, Staso PJ, Choi L, Peterson JF, Phillips EJ. DrugWAS: Leveraging drug-wide association studies to facilitate drug repurposing for COVID-19. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2021:2021.02.04.21251169. [PMID: 33564788 PMCID: PMC7872383 DOI: 10.1101/2021.02.04.21251169] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
Importance: There is an unprecedented need to rapidly identify safe and effective treatments for the novel coronavirus disease 2019 (COVID-19). Objective: To systematically investigate if any of the available drugs in Electronic Health Record (EHR), including prescription drugs and dietary supplements, can be repurposed as potential treatment for COVID-19. Design, Setting, and Participants: Based on a retrospective cohort analysis of EHR data, drug-wide association studies (DrugWAS) were performed on COVID-19 patients at Vanderbilt University Medical Center (VUMC). For each drug study, multivariable logistic regression with overlap weighting using propensity score was applied to estimate the effect of drug exposure on COVID-19 disease outcomes. Exposures: Patient exposure to a drug during 1-year prior to the pandemic and COVID-19 diagnosis was chosen as exposure of interest. Natural language processing was employed to extract drug information from clinical notes, in addition to the prescription drug data available in structured format. Main Outcomes and Measures: All-cause of death was selected as primary outcome. Hospitalization, admission to the intensive care unit (ICU), and need for mechanical ventilation were identified as secondary outcomes. Results: The study included 7,768 COVID-19 patients, of which 509 (6.55%) were hospitalized, 82 (1.06%) were admitted to ICU, 64 (0.82%) received mechanical ventilation, and 90 (1.16%) died. Overall, 15 drugs were significantly associated with decreased COVID-19 severity. Previous exposure to either Streptococcus pneumoniae vaccines (adjusted odds ratio [OR], 0.38; 95% CI, 0.14-0.98), diphtheria toxoid vaccine (OR, 0.39; 95% CI, 0.15-0.98), and tetanus toxoid vaccine (OR, 0.39; 95% CI, 0.15-0.98) were significantly associated with a decreased risk of death (primary outcome). Secondary analyses identified several other significant associations showing lower risk for COVID-19 outcomes: 2 vaccines (acellular pertussis, Streptococcus pneumoniae), 3 dietary supplements (turmeric extract, flaxseed extract, omega-3 fatty acids), methylprednisolone acetate, pseudoephedrine, ethinyl estradiol, estradiol, ibuprofen, and fluticasone. Conclusions and Relevance: This cohort study leveraged EHR data to identify a list of drugs that could be repurposed to improve COVID-19 outcomes. Further randomized clinical trials are needed to investigate the efficacy of the proposed drugs.
Collapse
Affiliation(s)
- Cosmin A. Bejan
- Department of Biomedical Informatics; Vanderbilt University Medical Center; Nashville, USA
| | - Katherine N. Cahill
- Department of Medicine; Division of Allergy, Pulmonary and Critical Care Medicine; Vanderbilt University Medical Center; Nashville, USA
| | - Patrick J. Staso
- Department of Medicine; Division of Allergy, Pulmonary and Critical Care Medicine; Vanderbilt University Medical Center; Nashville, USA
| | - Leena Choi
- Department of Biostatistics; Vanderbilt University Medical Center; Nashville, USA
| | - Josh F. Peterson
- Department of Biomedical Informatics; Vanderbilt University Medical Center; Nashville, USA
- Department of Medicine; Vanderbilt University Medical Center; Nashville, USA
| | - Elizabeth J. Phillips
- Department of Pathology, Microbiology and Immunology; Vanderbilt University Medical Center; Nashville, USA
- Department of Medicine; Division of Infectious Diseases; Vanderbilt University Medical Center; Nashville, USA
- Department of Pharmacology; Vanderbilt University Medical Center; Nashville, USA
| |
Collapse
|
6
|
Performance of a Natural Language Processing Method to Extract Stone Composition From the Electronic Health Record. Urology 2019; 132:56-62. [PMID: 31310771 DOI: 10.1016/j.urology.2019.07.007] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2019] [Revised: 07/03/2019] [Accepted: 07/05/2019] [Indexed: 12/24/2022]
Abstract
OBJECTIVES To demonstrate the utility of a natural language processing (NLP) algorithm for mining kidney stone composition in a large-scale electronic health records (EHR) repository. METHODS We developed StoneX, a pattern-matching method for extracting kidney stone composition information from clinical notes. We trained the extraction algorithm on manually annotated text mentions of calcium oxalate monohydrate, calcium oxalate dihydrate, hydroxyapatite, brushite, uric acid, and struvite stones. We employed StoneX to identify patients with kidney stone composition data and mine >125 million notes from our institutional EHR. Analyses performed on the extracted patients included stone type conversions over time, survival analysis from a second stone surgery, and disease associations by stone composition to validate the phenotyping method against known associations. RESULTS The NLP algorithm identified 45,235 text mentions corresponding to 11,585 patients. Overall, the system achieved positive predictive value >90% for calcium oxalate monohydrate, calcium oxalate dihydrate, hydroxyapatite, brushite, and struvite; except for uric acid (positive predictive value = 87.5%). Survival analysis from a second stone surgery showed statistically significant differences among stone types (P = .03). Several phenotype associations were found: uric acid-type 2 diabetes (odds ratio, OR = 2.69, 95% confidence intervals, CI = 1.91-3.79), struvite-neurogenic bladder (OR = 12.27, 95% CI = 4.33-34.79), struvite-urinary tract infection (OR = 7.36, 95% CI = 3.01-17.99), hydroxyapatite-pulmonary collapse (OR = 3.67, 95% CI = 2.10-6.42), hydroxyapatite-neurogenic bladder (OR = 5.23, 95% CI = 2.05-13.36), brushite-calcium metabolism disorder (OR = 4.59, 95% CI = 2.14-9.81), and brushite-hypercalcemia (OR = 4.09, 95% CI = 1.90-8.80). CONCLUSION NLP extraction of kidney stone composition from large-scale EHRs is feasible with high precision, enabling high-throughput epidemiological studies of kidney stone disease. These tools will enable high fidelity kidney stone research from the EHR.
Collapse
|
7
|
Bejan CA, Angiolillo J, Conway D, Nash R, Shirey-Rice JK, Lipworth L, Cronin RM, Pulley J, Kripalani S, Barkin S, Johnson KB, Denny JC. Mining 100 million notes to find homelessness and adverse childhood experiences: 2 case studies of rare and severe social determinants of health in electronic health records. J Am Med Inform Assoc 2018; 25:61-71. [PMID: 29016793 PMCID: PMC6080810 DOI: 10.1093/jamia/ocx059] [Citation(s) in RCA: 62] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2016] [Revised: 04/22/2017] [Accepted: 05/10/2017] [Indexed: 01/25/2023] Open
Abstract
Objective Understanding how to identify the social determinants of health from electronic health records (EHRs) could provide important insights to understand health or disease outcomes. We developed a methodology to capture 2 rare and severe social determinants of health, homelessness and adverse childhood experiences (ACEs), from a large EHR repository. Materials and Methods We first constructed lexicons to capture homelessness and ACE phenotypic profiles. We employed word2vec and lexical associations to mine homelessness-related words. Next, using relevance feedback, we refined the 2 profiles with iterative searches over 100 million notes from the Vanderbilt EHR. Seven assessors manually reviewed the top-ranked results of 2544 patient visits relevant for homelessness and 1000 patients relevant for ACE. Results word2vec yielded better performance (area under the precision-recall curve [AUPRC] of 0.94) than lexical associations (AUPRC = 0.83) for extracting homelessness-related words. A comparative study of searches for the 2 phenotypes revealed a higher performance achieved for homelessness (AUPRC = 0.95) than ACE (AUPRC = 0.79). A temporal analysis of the homeless population showed that the majority experienced chronic homelessness. Most ACE patients suffered sexual (70%) and/or physical (50.6%) abuse, with the top-ranked abuser keywords being "father" (21.8%) and "mother" (15.4%). Top prevalent associated conditions for homeless patients were lack of housing (62.8%) and tobacco use disorder (61.5%), while for ACE patients it was mental disorders (36.6%-47.6%). Conclusion We provide an efficient solution for mining homelessness and ACE information from EHRs, which can facilitate large clinical and genetic studies of these social determinants of health.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Robert M Cronin
- Department of Biomedical Informatics
- Department of Medicine
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA
| | | | | | - Shari Barkin
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Kevin B Johnson
- Department of Biomedical Informatics
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Joshua C Denny
- Department of Biomedical Informatics
- Department of Medicine
| |
Collapse
|
8
|
Yim WW, Kwan SW, Yetisgen M. Classifying tumor event attributes in radiology reports. J Assoc Inf Sci Technol 2017. [DOI: 10.1002/asi.23937] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Affiliation(s)
- Wen-wai Yim
- Palo Alto Veterans Affairs, Biomedical Informatics Research, Stanford University, 1265 Welch Road; Stanford CA 94305
| | - Sharon W. Kwan
- Department of Radiology; Interventional Radiology Section, University of Washington Medical Center, 1959 NE Pacific Street; Seattle WA 98195 USA
| | - Meliha Yetisgen
- Biomedical and Health Informatics, Linguistics; University of Washington, Box 358047; Seattle WA 98195 USA
| |
Collapse
|
9
|
Yetisgen M, Vanderwende L. Automatic Identification of Substance Abuse from Social History in Clinical Text. Artif Intell Med 2017. [DOI: 10.1007/978-3-319-59758-4_18] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
|
10
|
Kumar V, Stubbs A, Shaw S, Uzuner Ö. Creation of a new longitudinal corpus of clinical narratives. J Biomed Inform 2015; 58 Suppl:S6-S10. [PMID: 26433122 PMCID: PMC4978168 DOI: 10.1016/j.jbi.2015.09.018] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2015] [Revised: 09/22/2015] [Accepted: 09/23/2015] [Indexed: 10/23/2022]
Abstract
The 2014 i2b2/UTHealth Natural Language Processing (NLP) shared task featured a new longitudinal corpus of 1304 records representing 296 diabetic patients. The corpus contains three cohorts: patients who have a diagnosis of coronary artery disease (CAD) in their first record, and continue to have it in subsequent records; patients who do not have a diagnosis of CAD in the first record, but develop it by the last record; patients who do not have a diagnosis of CAD in any record. This paper details the process used to select records for this corpus and provides an overview of novel research uses for this corpus. This corpus is the only annotated corpus of longitudinal clinical narratives currently available for research to the general research community.
Collapse
Affiliation(s)
- Vishesh Kumar
- Dartmouth-Hitchcock Medical Center, Division of Cardiology, Lebanon, NH, USA
| | - Amber Stubbs
- School of Library and Information Science, Simmons College, Boston, MA, USA.
| | - Stanley Shaw
- Harvard Medical School, Boston, MA 02115, USA; Center for Systems Biology, Massachusetts General Hospital, Boston, MA 02114, USA
| | - Özlem Uzuner
- Department of Information Studies, State University of New York at Albany, Albany, NY, USA
| |
Collapse
|
11
|
Uzuner Ö, Stubbs A. Practical applications for natural language processing in clinical research: The 2014 i2b2/UTHealth shared tasks. J Biomed Inform 2015; 58 Suppl:S1-S5. [PMID: 26515500 PMCID: PMC4978169 DOI: 10.1016/j.jbi.2015.10.007] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2015] [Revised: 10/08/2015] [Accepted: 10/14/2015] [Indexed: 12/29/2022]
Affiliation(s)
- Özlem Uzuner
- Department of Information Studies, State University of New York at Albany, Albany, NY, USA.
| | - Amber Stubbs
- School of Library and Information Science, Simmons College, Boston, MA, USA.
| |
Collapse
|
12
|
Bejan CA, Wei WQ, Denny JC. Assessing the role of a medication-indication resource in the treatment relation extraction from clinical text. J Am Med Inform Assoc 2014; 22:e162-76. [PMID: 25336593 DOI: 10.1136/amiajnl-2014-002954] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2014] [Accepted: 09/27/2014] [Indexed: 12/30/2022] Open
Abstract
OBJECTIVE To evaluate the contribution of the MEDication Indication (MEDI) resource and SemRep for identifying treatment relations in clinical text. MATERIALS AND METHODS We first processed clinical documents with SemRep to extract the Unified Medical Language System (UMLS) concepts and the treatment relations between them. Then, we incorporated MEDI into a simple algorithm that identifies treatment relations between two concepts if they match a medication-indication pair in this resource. For a better coverage, we expanded MEDI using ontology relationships from RxNorm and UMLS Metathesaurus. We also developed two ensemble methods, which combined the predictions of SemRep and the MEDI algorithm. We evaluated our selected methods on two datasets, a Vanderbilt corpus of 6864 discharge summaries and the 2010 Informatics for Integrating Biology and the Bedside (i2b2)/Veteran's Affairs (VA) challenge dataset. RESULTS The Vanderbilt dataset included 958 manually annotated treatment relations. A double annotation was performed on 25% of relations with high agreement (Cohen's κ = 0.86). The evaluation consisted of comparing the manual annotated relations with the relations identified by SemRep, the MEDI algorithm, and the two ensemble methods. On the first dataset, the best F1-measure results achieved by the MEDI algorithm and the union of the two resources (78.7 and 80, respectively) were significantly higher than the SemRep results (72.3). On the second dataset, the MEDI algorithm achieved better precision and significantly lower recall values than the best system in the i2b2 challenge. The two systems obtained comparable F1-measure values on the subset of i2b2 relations with both arguments in MEDI. CONCLUSIONS Both SemRep and MEDI can be used to extract treatment relations from clinical text. Knowledge-based extraction with MEDI outperformed use of SemRep alone, but superior performance was achieved by integrating both systems. The integration of knowledge-based resources such as MEDI into information extraction systems such as SemRep and the i2b2 relation extractors may improve treatment relation extraction from clinical text.
Collapse
Affiliation(s)
- Cosmin Adrian Bejan
- Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee, USA
| | - Wei-Qi Wei
- Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee, USA
| | - Joshua C Denny
- Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee, USA Department of Medicine, Vanderbilt University, Nashville, Tennessee, USA
| |
Collapse
|
13
|
Hanauer DA, Saeed M, Zheng K, Mei Q, Shedden K, Aronson AR, Ramakrishnan N. Applying MetaMap to Medline for identifying novel associations in a large clinical dataset: a feasibility analysis. J Am Med Inform Assoc 2014; 21:925-37. [PMID: 24928177 PMCID: PMC4147617 DOI: 10.1136/amiajnl-2014-002767] [Citation(s) in RCA: 19] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2014] [Revised: 05/23/2014] [Accepted: 05/27/2014] [Indexed: 02/07/2023] Open
Abstract
OBJECTIVE We describe experiments designed to determine the feasibility of distinguishing known from novel associations based on a clinical dataset comprised of International Classification of Disease, V.9 (ICD-9) codes from 1.6 million patients by comparing them to associations of ICD-9 codes derived from 20.5 million Medline citations processed using MetaMap. Associations appearing only in the clinical dataset, but not in Medline citations, are potentially novel. METHODS Pairwise associations of ICD-9 codes were independently identified in both the clinical and Medline datasets, which were then compared to quantify their degree of overlap. We also performed a manual review of a subset of the associations to validate how well MetaMap performed in identifying diagnoses mentioned in Medline citations that formed the basis of the Medline associations. RESULTS The overlap of associations based on ICD-9 codes in the clinical and Medline datasets was low: only 6.6% of the 3.1 million associations found in the clinical dataset were also present in the Medline dataset. Further, a manual review of a subset of the associations that appeared in both datasets revealed that co-occurring diagnoses from Medline citations do not always represent clinically meaningful associations. DISCUSSION Identifying novel associations derived from large clinical datasets remains challenging. Medline as a sole data source for existing knowledge may not be adequate to filter out widely known associations. CONCLUSIONS In this study, novel associations were not readily identified. Further improvements in accuracy and relevance for tools such as MetaMap are needed to realize their expected utility.
Collapse
Affiliation(s)
- David A Hanauer
- Department of Pediatrics, University of Michigan Medical School, Ann Arbor, Michigan, USA
| | - Mohammed Saeed
- Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, Michigan, USA
| | - Kai Zheng
- Department of Health Management and Policy, University of Michigan School of Public Health, Ann Arbor, Michigan, USA
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Qiaozhu Mei
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
- Department of Electronic Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan, USA
| | - Kerby Shedden
- Center for Statistical Consultation and Research, University of Michigan, Ann Arbor, Michigan, USA
| | - Alan R Aronson
- Lister Hill Center, National Library of Medicine, Bethesda, Maryland, USA
| | - Naren Ramakrishnan
- Department of Computer Science, Discovery Analytics Center, Virginia Tech, Arlington, Virginia, USA
| |
Collapse
|