1
|
Vasan V, Cheng CP, Fan CJ, Lerner DK, Pascual K, Iloreta AM, Babu SC, Cosetti MK. Gender Differences in Letters of Recommendations and Personal Statements for Neurotology Fellowship over 10 Years: A Deep Learning Linguistic Analysis. Otol Neurotol 2024; 45:827-832. [PMID: 39052892 DOI: 10.1097/mao.0000000000004265] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/27/2024]
Abstract
OBJECTIVE Personal statements (PSs) and letters of recommendation (LORs) are critical components of the neurotology fellowship application process but can be subject to implicit biases. This study evaluated general and deep learning linguistic differences between the applicant genders over a 10-year span. STUDY DESIGN Retrospective cohort. SETTING Two institutions. MAIN OUTCOME MEASURES PSs and LORs were collected from 2014 to 2023 from two institutions. The Valence Aware Dictionary and Sentiment Reasoner (VADER) natural language processing (NLP) package was used to compare the positive or negative sentiment in LORs and PSs. Next, the deep learning tool, Empath, categorized the text into scores, and Wilcoxon rank sum tests were performed for comparisons between applicant gender. RESULTS Among 177 applicants over 10 years, 120 were males and 57 were females. There were no differences in word count or VADER sentiment scores between genders for both LORs and PSs. However, among Empath sentiment categories, male applicants had more words of trust ( p = 0.03) and leadership ( p = 0.002) in LORs. Temporally, the trends show a consistently higher VADER sentiment and Empath "trust" and "leader" in male LORs from 2014 to 2019, after which there was no statistical significance in sentiment scores between genders, and females even have higher scores of trust and leadership in 2023. CONCLUSIONS Linguistic content overall favored male applicants because they were more frequently described as trustworthy and leaders. However, the temporal analysis of linguistic differences between male and female applicants found an encouraging trend suggesting a reduction of gender bias in recent years, mirroring an increased composition of women in neurotology over time.
Collapse
Affiliation(s)
- Vikram Vasan
- Department of Otolaryngology-Head and Neck Surgery, Icahn School of Medicine at Mount Sinai, New York, New York
| | - Christopher P Cheng
- Department of Otolaryngology-Head and Neck Surgery, Icahn School of Medicine at Mount Sinai, New York, New York
| | | | | | - Karen Pascual
- Department of Otolaryngology-Head and Neck Surgery, Icahn School of Medicine at Mount Sinai, New York, New York
| | - Alfred Marc Iloreta
- Department of Otolaryngology-Head and Neck Surgery, Icahn School of Medicine at Mount Sinai, New York, New York
| | | | - Maura K Cosetti
- Department of Otolaryngology-Head and Neck Surgery, Icahn School of Medicine at Mount Sinai, New York, New York
| |
Collapse
|
2
|
Nolan VJ, Balch JA, Baskaran NP, Shickel B, Efron PA, Upchurch GR, Bihorac A, Tignanelli CJ, Moseley RE, Loftus TJ. Incorporating Patient Values in Large Language Model Recommendations for Surrogate and Proxy Decisions. Crit Care Explor 2024; 6:e1131. [PMID: 39132980 PMCID: PMC11321752 DOI: 10.1097/cce.0000000000001131] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/13/2024] Open
Abstract
BACKGROUND Surrogates, proxies, and clinicians making shared treatment decisions for patients who have lost decision-making capacity often fail to honor patients' wishes, due to stress, time pressures, misunderstanding patient values, and projecting personal biases. Advance directives intend to align care with patient values but are limited by low completion rates and application to only a subset of medical decisions. Here, we investigate the potential of large language models (LLMs) to incorporate patient values in supporting critical care clinical decision-making for incapacitated patients in a proof-of-concept study. METHODS We simulated text-based scenarios for 50 decisionally incapacitated patients for whom a medical condition required imminent clinical decisions regarding specific interventions. For each patient, we also simulated five unique value profiles captured using alternative formats: numeric ranking questionnaires, text-based questionnaires, and free-text narratives. We used pre-trained generative LLMs for two tasks: 1) text extraction of the treatments under consideration and 2) prompt-based question-answering to generate a recommendation in response to the scenario information, extracted treatment, and patient value profiles. Model outputs were compared with adjudications by three domain experts who independently evaluated each scenario and decision. RESULTS AND CONCLUSIONS Automated extractions of the treatment in question were accurate for 88% (n = 44/50) of scenarios. LLM treatment recommendations received an average Likert score by the adjudicators of 3.92 of 5.00 (five being best) across all patients for being medically plausible and reasonable treatment recommendations, and 3.58 of 5.00 for reflecting the documented values of the patient. Scores were highest when patient values were captured as short, unstructured, and free-text narratives based on simulated patient profiles. This proof-of-concept study demonstrates the potential for LLMs to function as support tools for surrogates, proxies, and clinicians aiming to honor the wishes and values of decisionally incapacitated patients.
Collapse
Affiliation(s)
| | - Jeremy A. Balch
- Department of Surgery, University of Florida, Gainesville, FL
| | | | | | - Philip A. Efron
- Department of Surgery, University of Florida, Gainesville, FL
| | | | - Azra Bihorac
- Department of Medicine, University of Florida, Gainesville, FL
| | | | - Ray E. Moseley
- College of Medicine, University of Florida, Gainesville, FL
| | - Tyler J. Loftus
- Department of Surgery, University of Florida, Gainesville, FL
| |
Collapse
|
3
|
Wang Z, Li Y, Zhai J, Yang S, Sun B, Liang P. Deep learning-based Raman spectroscopy qualitative analysis algorithm: A convolutional neural network and transformer approach. Talanta 2024; 275:126138. [PMID: 38677164 DOI: 10.1016/j.talanta.2024.126138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2024] [Revised: 04/16/2024] [Accepted: 04/19/2024] [Indexed: 04/29/2024]
Abstract
Raman spectroscopy is a general and non-destructive detection technique that can obtain detailed information of the chemical structure of materials. In the past, when using chemometric algorithms to analyze the Raman spectra of mixtures, the challenges of complex spectral overlap and noise often limited the accurate identification of components. The emergence of deep learning has introduced a novel approach to qualitative analysis of mixed Raman spectra. In this paper, we propose a deep learning-based Raman spectroscopy qualitative analysis algorithm (RST) by borrowing the ideas of convolutional neural network and Transformer. By transforming the Raman spectrum into 64 word vectors, the contribution weights of each word vector to the components are obtained. For the 75 spectral data used for validation, the positive identification rate can reach 100.00 %, the recall rate can reach 99.3 %, the average identification score can reach 9.51, and it is applicable to the fields of Raman and surface-enhanced Raman spectroscopy. Furthermore, compared with traditional CNN models, RST has excellent accuracy and robustness in identifying components in complex mixtures. The model's interpretability has been enhanced, aiding in a deeper understanding of spectroscopic learning patterns for future analysis of more complex mixtures.
Collapse
Affiliation(s)
- Zilong Wang
- College of Optical and Electronic Technology, China Jiliang University, Hangzhou, 310018, China; Xiamen Palantier Technology Co., Ltd., Xiamen, 361000, China
| | - Yunfeng Li
- College of Information Engineering, China Jiliang University, Hangzhou, 310018, China
| | - Jinglei Zhai
- School of Electrical and Information Engineering, Tianjin University, No. 92, Weijin Road, Nankai District, Tianjin, 300072, China
| | - Siwei Yang
- College of Optical and Electronic Technology, China Jiliang University, Hangzhou, 310018, China
| | - Biao Sun
- School of Electrical and Information Engineering, Tianjin University, No. 92, Weijin Road, Nankai District, Tianjin, 300072, China
| | - Pei Liang
- College of Optical and Electronic Technology, China Jiliang University, Hangzhou, 310018, China; Xiamen Palantier Technology Co., Ltd., Xiamen, 361000, China.
| |
Collapse
|
4
|
Yoon W, Chen S, Gao Y, Zhao Z, Dligach D, Bitterman DS, Afshar M, Miller T. LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction for Language Models. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.03.26.24304920. [PMID: 38585973 PMCID: PMC10996733 DOI: 10.1101/2024.03.26.24304920] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Objective The application of Natural Language Processing (NLP) in the clinical domain is important due to the rich unstructured information in clinical documents, which often remains inaccessible in structured data. When applying NLP methods to a certain domain, the role of benchmark datasets is crucial as benchmark datasets not only guide the selection of best-performing models but also enable the assessment of the reliability of the generated outputs. Despite the recent availability of language models (LMs) capable of longer context, benchmark datasets targeting long clinical document classification tasks are absent. Materials and Methods To address this issue, we propose LCD benchmark, a benchmark for the task of predicting 30-day out-of-hospital mortality using discharge notes of MIMIC-IV and statewide death data. We evaluated this benchmark dataset using baseline models, from bag-of-words and CNN to instruction-tuned large language models. Additionally, we provide a comprehensive analysis of the model outputs, including manual review and visualization of model weights, to offer insights into their predictive capabilities and limitations. Results and Discussion Baseline models showed 28.9% for best-performing supervised models and 32.2% for GPT-4 in F1-metrics. Notes in our dataset have a median word count of 1687. Our analysis of the model outputs showed that our dataset is challenging for both models and human experts, but the models can find meaningful signals from the text. Conclusion We expect our LCD benchmark to be a resource for the development of advanced supervised models, or prompting methods, tailored for clinical text. The benchmark dataset is available at https://github.com/Machine-Learning-for-Medical-Language/long-clinical-doc.
Collapse
|
5
|
Wieland-Jorna Y, van Kooten D, Verheij RA, de Man Y, Francke AL, Oosterveld-Vlug MG. Natural language processing systems for extracting information from electronic health records about activities of daily living. A systematic review. JAMIA Open 2024; 7:ooae044. [PMID: 38798774 PMCID: PMC11126158 DOI: 10.1093/jamiaopen/ooae044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 03/21/2024] [Accepted: 05/07/2024] [Indexed: 05/29/2024] Open
Abstract
Objective Natural language processing (NLP) can enhance research on activities of daily living (ADL) by extracting structured information from unstructured electronic health records (EHRs) notes. This review aims to give insight into the state-of-the-art, usability, and performance of NLP systems to extract information on ADL from EHRs. Materials and Methods A systematic review was conducted based on searches in Pubmed, Embase, Cinahl, Web of Science, and Scopus. Studies published between 2017 and 2022 were selected based on predefined eligibility criteria. Results The review identified 22 studies. Most studies (65%) used NLP for classifying unstructured EHR data on 1 or 2 ADL. Deep learning, combined with a ruled-based method or machine learning, was the approach most commonly used. NLP systems varied widely in terms of the pre-processing and algorithms. Common performance evaluation methods were cross-validation and train/test datasets, with F1, precision, and sensitivity as the most frequently reported evaluation metrics. Most studies reported relativity high overall scores on the evaluation metrics. Discussion NLP systems are valuable for the extraction of unstructured EHR data on ADL. However, comparing the performance of NLP systems is difficult due to the diversity of the studies and challenges related to the dataset, including restricted access to EHR data, inadequate documentation, lack of granularity, and small datasets. Conclusion This systematic review indicates that NLP is promising for deriving information on ADL from unstructured EHR notes. However, what the best-performing NLP system is, depends on characteristics of the dataset, research question, and type of ADL.
Collapse
Affiliation(s)
- Yvonne Wieland-Jorna
- Netherlands Institute for Health Services Research (Nivel), Utrecht, Postbus 1568, 3500 BN, The Netherlands
- Tranzo, School of Social Sciences and Behavioural Research, Tilburg University, Tilburg, Postbus 90153, 5000 LE, The Netherlands
| | - Daan van Kooten
- Netherlands Institute for Health Services Research (Nivel), Utrecht, Postbus 1568, 3500 BN, The Netherlands
| | - Robert A Verheij
- Netherlands Institute for Health Services Research (Nivel), Utrecht, Postbus 1568, 3500 BN, The Netherlands
- Tranzo, School of Social Sciences and Behavioural Research, Tilburg University, Tilburg, Postbus 90153, 5000 LE, The Netherlands
| | - Yvonne de Man
- Netherlands Institute for Health Services Research (Nivel), Utrecht, Postbus 1568, 3500 BN, The Netherlands
| | - Anneke L Francke
- Netherlands Institute for Health Services Research (Nivel), Utrecht, Postbus 1568, 3500 BN, The Netherlands
- Department of Public and Occupational Health, Location Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, Postbus 7057, 1007 MB, The Netherlands
| | - Mariska G Oosterveld-Vlug
- Netherlands Institute for Health Services Research (Nivel), Utrecht, Postbus 1568, 3500 BN, The Netherlands
| |
Collapse
|
6
|
Ramachandran V, Jairath N, Cheraghlou S, Pahalyants V. Revolutionizing dermatology residency: artificial intelligence for knowledge and clinical milestones assessment. Clin Exp Dermatol 2024; 49:732-733. [PMID: 37715693 DOI: 10.1093/ced/llad324] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Accepted: 11/29/2023] [Indexed: 09/18/2023]
Abstract
Artificial intelligence can be adapted and used to augment resident education in order to assess clinical milestones in real-time, providing powerful insights for educators and trainees alike. Further research on integrating such technology into our models of learning fosters the growth of our specialty and trainees’ aptitudes.
Collapse
Affiliation(s)
| | - Neil Jairath
- Department of Dermatology, New York University, New York, NY, USA
| | | | | |
Collapse
|
7
|
Iscoe M, Socrates V, Gilson A, Chi L, Li H, Huang T, Kearns T, Perkins R, Khandjian L, Taylor RA. Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models. Acad Emerg Med 2024; 31:599-610. [PMID: 38567658 DOI: 10.1111/acem.14883] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 01/24/2024] [Accepted: 01/24/2024] [Indexed: 04/04/2024]
Abstract
BACKGROUND Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms. OBJECTIVES This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes. METHODS The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level. RESULTS A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model. CONCLUSIONS The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.
Collapse
Affiliation(s)
- Mark Iscoe
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA
| | - Vimig Socrates
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA
- Program of Computational Biology and Bioinformatics, Yale University, New Haven, Connecticut, USA
| | - Aidan Gilson
- Yale School of Medicine, New Haven, Connecticut, USA
| | - Ling Chi
- Department of Biostatistics, Yale School of Public Health, New Haven, Connecticut, USA
| | - Huan Li
- Program of Computational Biology and Bioinformatics, Yale University, New Haven, Connecticut, USA
| | - Thomas Huang
- Yale School of Medicine, New Haven, Connecticut, USA
| | - Thomas Kearns
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - Rachelle Perkins
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - Laura Khandjian
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - R Andrew Taylor
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA
| |
Collapse
|
8
|
Abu-Ashour W, Emil S, Poenaru D. Using Artificial Intelligence to Label Free-Text Operative and Ultrasound Reports for Grading Pediatric Appendicitis. J Pediatr Surg 2024; 59:783-790. [PMID: 38383177 DOI: 10.1016/j.jpedsurg.2024.01.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/11/2024] [Accepted: 01/22/2024] [Indexed: 02/23/2024]
Abstract
PURPOSE Data science approaches personalizing pediatric appendicitis management are hampered by small datasets and unstructured electronic medical records (EMR). Artificial intelligence (AI) chatbots based on large language models can structure free-text EMR data. We compare data extraction quality between ChatGPT-4 and human data collectors. METHODS To train AI models to grade pediatric appendicitis preoperatively, several data collectors extracted detailed preoperative and operative data from 2100 children operated for acute appendicitis. Collectors were trained for the task based on satisfactory Kappa scores. ChatGPT-4 was prompted to structure free text from 103 random anonymized ultrasound and operative records in the dataset using the set variables and coding options, and to estimate appendicitis severity grade from the operative report. A pediatric surgeon then adjudicated all data, identifying errors in each method. RESULTS Within the 44 ultrasound (42.7%) and 32 operative reports (31.1%) discordant in at least one field, 98% of the errors were found in the manual data extraction. The appendicitis grade was erroneously assigned manually in 29 patients (28.2%), and by ChatGPT-4 in 3 (2.9%). Across datasets, the use of the AI chatbot was able to avoid misclassification in 59.2% of the records including both reports and extracted data approximately 40 times faster. CONCLUSION AI chatbot significantly outperformed manual data extraction in accuracy for ultrasound and operative reports, and correctly assigned the appendicitis grade. While wider validation is required and data safety concerns must be addressed, these AI tools show significant promise in improving the accuracy and efficiency of research data collection. LEVELS OF EVIDENCE Level III.
Collapse
Affiliation(s)
- Waseem Abu-Ashour
- Harvey E. Beardmore Division of Pediatric Surgery, The Montreal Children's Hospital, McGill University Health Centre, Montreal, Quebec, Canada; McGill University Health Centre Research Institute, Montreal, Quebec, Canada.
| | - Sherif Emil
- Harvey E. Beardmore Division of Pediatric Surgery, The Montreal Children's Hospital, McGill University Health Centre, Montreal, Quebec, Canada; McGill University Health Centre Research Institute, Montreal, Quebec, Canada
| | - Dan Poenaru
- Harvey E. Beardmore Division of Pediatric Surgery, The Montreal Children's Hospital, McGill University Health Centre, Montreal, Quebec, Canada; McGill University Health Centre Research Institute, Montreal, Quebec, Canada
| |
Collapse
|
9
|
Böhringer D, Angelova P, Fuhrmann L, Zimmermann J, Schargus M, Eter N, Reinhard T. Automatic inference of ICD-10 codes from German ophthalmologic physicians' letters using natural language processing. Sci Rep 2024; 14:9035. [PMID: 38641674 PMCID: PMC11031573 DOI: 10.1038/s41598-024-59926-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Accepted: 04/16/2024] [Indexed: 04/21/2024] Open
Abstract
Physicians' letters are the optimal source of diagnoses for registries. However, most registries demand for diagnosis codes such as ICD-10. We herein describe an algorithm that infers ICD-10 codes from German ophthalmologic physicians' letters. We assess the method in three German eye hospitals. Our algorithm is based on the nearest-neighbor method as well as on a large thesaurus for ICD-10 codes. This thesaurus was embedded into a Word2Vec space created from anonymized physicians' reports of the first hospital. For evaluation, each of the three hospitals sent all diagnoses taken from 100 letters. The inferred ICD-10 codes were evaluated for correctness by the senders. A total of 3332 natural language terms had been sent in (812 hospital one, 1473 hospital two, 1047 hospital three). A total of 526 non-diagnoses were excluded upfront. 2806 ICD-10 codes were inferred (771 hospital one, 1226 hospital two, 809 hospital three). In the first hospital, 98% were fully correct and 99% correct at the level of the superordinate disease concept. The percentages in hospital two were 69% and 86%. The respective numbers for hospital three were 69% and 91%. Our simple method is capable of inferring ICD-10 codes for German natural language diagnoses, especially when the embedding space has been built with physicians' letters from the same hospital. The method may yield sufficient accuracy for many tasks in the multi-centric setting and can easily be adapted to other languages/specialities.
Collapse
Affiliation(s)
- D Böhringer
- Eye Center of the University Hospital Freiburg, Medical Faculty of the Albert-Ludwigs-University Freiburg, Freiburg, Germany.
| | - P Angelova
- Eye Center of the University Hospital Freiburg, Medical Faculty of the Albert-Ludwigs-University Freiburg, Freiburg, Germany
| | - L Fuhrmann
- Department of Ophthalmology, Asklepios Hospital Nord-Heidberg, Hamburg, Germany
| | - J Zimmermann
- Department of Ophthalmology, Medical Center, University of Münster, Münster, Germany
| | - M Schargus
- Department of Ophthalmology, Asklepios Hospital Nord-Heidberg, Hamburg, Germany
| | - N Eter
- Department of Ophthalmology, Medical Center, University of Münster, Münster, Germany
| | - T Reinhard
- Eye Center of the University Hospital Freiburg, Medical Faculty of the Albert-Ludwigs-University Freiburg, Freiburg, Germany
| |
Collapse
|
10
|
Seng EC, Mehdipour S, Simpson S, Gabriel RA. Tracking persistent postoperative opioid use: a proof-of-concept study demonstrating a use case for natural language processing. Reg Anesth Pain Med 2024; 49:241-247. [PMID: 37419509 DOI: 10.1136/rapm-2023-104629] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 06/24/2023] [Indexed: 07/09/2023]
Abstract
BACKGROUND Large language models have been gaining tremendous popularity since the introduction of ChatGPT in late 2022. Perioperative pain providers should leverage natural language processing (NLP) technology and explore pertinent use cases to improve patient care. One example is tracking persistent postoperative opioid use after surgery. Since much of the relevant data may be 'hidden' within unstructured clinical text, NLP models may prove to be advantageous. The primary objective of this proof-of-concept study was to demonstrate the ability of an NLP engine to review clinical notes and accurately identify patients who had persistent postoperative opioid use after major spine surgery. METHODS Clinical documents from all patients that underwent major spine surgery during July 2015-August 2021 were extracted from the electronic health record. The primary outcome was persistent postoperative opioid use, defined as continued use of opioids greater than or equal to 3 months after surgery. This outcome was ascertained via manual clinician review from outpatient spine surgery follow-up notes. An NLP engine was applied to these notes to ascertain the presence of persistent opioid use-this was then compared with results from clinician manual review. RESULTS The final study sample consisted of 965 patients, in which 705 (73.1%) were determined to have persistent opioid use following surgery. The NLP engine correctly determined the patients' opioid use status in 92.9% of cases, in which it correctly identified persistent opioid use in 95.6% of cases and no persistent opioid use in 86.1% of cases. DISCUSSION Access to unstructured data within the perioperative history can contextualize patients' opioid use and provide further insight into the opioid crisis, while at the same time improve care directly at the patient level. While these goals are in reach, future work is needed to evaluate how to best implement NLP within different healthcare systems for use in clinical decision support.
Collapse
Affiliation(s)
- Eri C Seng
- Division of Perioperative Informatics, Department of Anesthesiology, University of California San Diego, La Jolla, California, USA
| | - Soraya Mehdipour
- Division of Perioperative Informatics, Department of Anesthesiology, University of California San Diego, La Jolla, California, USA
| | - Sierra Simpson
- Division of Perioperative Informatics, Department of Anesthesiology, University of California San Diego, La Jolla, California, USA
| | - Rodney A Gabriel
- Division of Perioperative Informatics, Department of Anesthesiology, University of California San Diego, La Jolla, California, USA
- Division of Regional Anesthesia, Department of Anesthesiology, University of California, San Diego, La Jolla, California, USA
| |
Collapse
|
11
|
Lequertier V, Wang T, Fondrevelle J, Augusto V, Polazzi S, Duclos A. Length of Stay Prediction With Standardized Hospital Data From Acute and Emergency Care Using a Deep Neural Network. Med Care 2024; 62:225-234. [PMID: 38345863 DOI: 10.1097/mlr.0000000000001975] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/10/2024]
Abstract
OBJECTIVE Length of stay (LOS) is an important metric for the organization and scheduling of care activities. This study sought to propose a LOS prediction method based on deep learning using widely available administrative data from acute and emergency care and compare it with other methods. PATIENTS AND METHODS All admissions between January 1, 2011 and December 31, 2019, at 6 university hospitals of the Hospices Civils de Lyon metropolis were included, leading to a cohort of 1,140,100 stays of 515,199 patients. Data included demographics, primary and associated diagnoses, medical procedures, the medical unit, the admission type, socio-economic factors, and temporal information. A model based on embeddings and a Feed-Forward Neural Network (FFNN) was developed to provide fine-grained LOS predictions per hospitalization step. Performances were compared with random forest and logistic regression, with the accuracy, Cohen kappa, and a Bland-Altman plot, through a 5-fold cross-validation. RESULTS The FFNN achieved an accuracy of 0.944 (CI: 0.937, 0.950) and a kappa of 0.943 (CI: 0.935, 0.950). For the same metrics, random forest yielded 0.574 (CI: 0.573, 0.575) and 0.602 (CI: 0.601, 0.603), respectively, and 0.352 (CI: 0.346, 0.358) and 0.414 (CI: 0.408, 0.422) for the logistic regression. The FFNN had a limit of agreement ranging from -2.73 to 2.67, which was better than random forest (-6.72 to 6.83) or logistic regression (-7.60 to 9.20). CONCLUSION The FFNN was better at predicting LOS than random forest or logistic regression. Implementing the FFNN model for routine acute care could be useful for improving the quality of patients' care.
Collapse
Affiliation(s)
- Vincent Lequertier
- Research on Healthcare Performance RESHAPE, Inserm U1290, Université Claude Bernard Lyon 1, Lyon, France
- INSA Lyon, Université Lumière Lyon 2, Université Claude Bernard Lyon 1, Université Jean Monnet Saint-Etienne, DISP UR4570, Villeurbanne, France
| | - Tao Wang
- Université Jean Monnet Saint-Etienne, INSA Lyon, Université Lumière Lyon 2, Université Claude Bernard Lyon 1, DISP UR4570, Roanne, France
| | - Julien Fondrevelle
- INSA Lyon, Université Lumière Lyon 2, Université Claude Bernard Lyon 1, Université Jean Monnet Saint-Etienne, DISP UR4570, Villeurbanne, France
| | - Vincent Augusto
- Mines Saint-Etienne, Univ. Clermont Auvergne, CNRS, CIS Center, Saint-Etienne, France
| | - Stéphanie Polazzi
- Research on Healthcare Performance RESHAPE, Inserm U1290, Université Claude Bernard Lyon 1, Lyon, France
- Department of Health Data, Lyon University Hospital, Lyon, France
| | - Antoine Duclos
- Research on Healthcare Performance RESHAPE, Inserm U1290, Université Claude Bernard Lyon 1, Lyon, France
- Department of Health Data, Lyon University Hospital, Lyon, France
| |
Collapse
|
12
|
Gu Z, He X, Yu P, Jia W, Yang X, Peng G, Hu P, Chen S, Chen H, Lin Y. Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model. Artif Intell Med 2024; 150:102822. [PMID: 38553162 DOI: 10.1016/j.artmed.2024.102822] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 01/28/2024] [Accepted: 02/21/2024] [Indexed: 04/02/2024]
Abstract
BACKGROUND Stroke is a prevalent disease with a significant global impact. Effective assessment of stroke severity is vital for an accurate diagnosis, appropriate treatment, and optimal clinical outcomes. The National Institutes of Health Stroke Scale (NIHSS) is a widely used scale for quantitatively assessing stroke severity. However, the current manual scoring of NIHSS is labor-intensive, time-consuming, and sometimes unreliable. Applying artificial intelligence (AI) techniques to automate the quantitative assessment of stroke on vast amounts of electronic health records (EHRs) has attracted much interest. OBJECTIVE This study aims to develop an automatic, quantitative stroke severity assessment framework through automating the entire NIHSS scoring process on Chinese clinical EHRs. METHODS Our approach consists of two major parts: Chinese clinical named entity recognition (CNER) with a domain-adaptive pre-trained large language model (LLM) and automated NIHSS scoring. To build a high-performing CNER model, we first construct a stroke-specific, densely annotated dataset "Chinese Stroke Clinical Records" (CSCR) from EHRs provided by our partner hospital, based on a stroke ontology that defines semantically related entities for stroke assessment. We then pre-train a Chinese clinical LLM coined "CliRoberta" through domain-adaptive transfer learning and construct a deep learning-based CNER model that can accurately extract entities directly from Chinese EHRs. Finally, an automated, end-to-end NIHSS scoring pipeline is proposed by mapping the extracted entities to relevant NIHSS items and values, to quantitatively assess the stroke severity. RESULTS Results obtained on a benchmark dataset CCKS2019 and our newly created CSCR dataset demonstrate the superior performance of our domain-adaptive pre-trained LLM and the CNER model, compared with the existing benchmark LLMs and CNER models. The high F1 score of 0.990 ensures the reliability of our model in accurately extracting the entities for the subsequent automatic NIHSS scoring. Subsequently, our automated, end-to-end NIHSS scoring approach achieved excellent inter-rater agreement (0.823) and intraclass consistency (0.986) with the ground truth and significantly reduced the processing time from minutes to a few seconds. CONCLUSION Our proposed automatic and quantitative framework for assessing stroke severity demonstrates exceptional performance and reliability through directly scoring the NIHSS from diagnostic notes in Chinese clinical EHRs. Moreover, this study also contributes a new clinical dataset, a pre-trained clinical LLM, and an effective deep learning-based CNER model. The deployment of these advanced algorithms can improve the accuracy and efficiency of clinical assessment, and help improve the quality, affordability and productivity of healthcare services.
Collapse
Affiliation(s)
- Zhanzhong Gu
- School of Electrical and Data Engineering, University of Technology Sydney, NSW, 2007, Australia.
| | - Xiangjian He
- School of Electrical and Data Engineering, University of Technology Sydney, NSW, 2007, Australia; School of Computer Science, University of Nottingham Ningbo China, Ningbo, China
| | - Ping Yu
- School of Computing and Information Technology, University of Wollongong, NSW, 2522, Australia
| | - Wenjing Jia
- School of Electrical and Data Engineering, University of Technology Sydney, NSW, 2007, Australia
| | - Xiguang Yang
- School of Electrical and Data Engineering, University of Technology Sydney, NSW, 2007, Australia
| | - Gang Peng
- Intergenepharm Pty Ltd, Sydney, NSW, 2000, Australia
| | - Penghui Hu
- Department of Oncology, The First Affiliated Hospital of Jinan University, Guangzhou, China
| | - Shiyan Chen
- Department of Neurology, The First Affiliated Hospital of Fujian Medical University, Fuzhou, China
| | - Hongjie Chen
- Department of Traditional Chinese Medicine, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Yiguang Lin
- Department of Traditional Chinese Medicine, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China; Department of Immuno-Oncology, The First Affiliated Hospital of Guangdong Pharmaceutical University, China; School of Life Sciences, University of Technology Sydney, NSW, 2007, Australia
| |
Collapse
|
13
|
Quistberg DA. Potential of artificial intelligence in injury prevention research and practice. Inj Prev 2024; 30:89-91. [PMID: 38307714 PMCID: PMC11003389 DOI: 10.1136/ip-2023-045203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 01/18/2024] [Indexed: 02/04/2024]
Abstract
There is increasing interest and use of artificial Intelligence algorithms and methods in biomedical research and practice, particularly as the technology has made significant advances in the past decade and become more accessible to more disciplines. This editorial briefly reviews this technology and its potential for injury prevention research and practice, proposing ways that it can be used to advance the discipline, as well as the potential pitfalls, concerns and biases that accompany it.
Collapse
Affiliation(s)
- D Alex Quistberg
- Urban Health Collaborative, Drexel University, Philadelphia, Pennsylvania, USA
- Environmental & Occupational Health, Drexel University, Philadelphia, Pennsylvania, USA
| |
Collapse
|
14
|
Goldenholz DM, Goldenholz SR, Habib S, Westover MB. Inductive reasoning with large language models: a simulated randomized controlled trial for epilepsy. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.03.18.24304493. [PMID: 38562831 PMCID: PMC10984041 DOI: 10.1101/2024.03.18.24304493] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Importance The analysis of electronic medical records at scale to learn from clinical experience is currently very challenging. The integration of artificial intelligence (AI), specifically foundational large language models (LLMs), into an analysis pipeline may overcome some of the current limitations of modest input sizes, inaccuracies, biases, and incomplete knowledge bases. Objective To explore the effectiveness of using an LLM for generating realistic clinical data and other LLMs for summarizing and synthesizing information in a model system, simulating a randomized clinical trial (RCT) in epilepsy to demonstrate the potential of inductive reasoning via medical chart review. Design An LLM-generated simulated RCT based on a RCT for treatment with an antiseizure medication, cenobamate, including a placebo arm and a full-strength drug arm, evaluated by an LLM-based pipeline versus a human reader. Setting Simulation based on realistic seizure diaries, treatment effects, reported symptoms and clinical notes generated by LLMs with multiple different neurologist writing styles. Participants Simulated cohort of 240 patients, divided 1:1 into placebo and drug arms. Intervention Utilization of LLMs for the generation of clinical notes and for the synthesis of data from these notes, aiming to evaluate the efficacy and safety of cenobamate in seizure control either with a human evaluator or AI-pipeline. Measures The AI and human analysis focused on identifying the number of seizures, symptom reports, and treatment efficacy, with statistical analysis comparing the 50%-responder rate and median percentage change between the placebo and drug arms, as well as side effect rates in each arm. Results AI closely mirrored human analysis, demonstrating the drug's efficacy with marginal differences (<3%) in identifying both drug efficacy and reported symptoms. Conclusions and Relevance This study showcases the potential of LLMs accurately simulate and analyze clinical trials. Significantly, it highlights the ability of LLMs to reconstruct essential trial elements, identify treatment effects, and recognize reported symptoms, within a realistic clinical framework. The findings underscore the relevance of LLMs in future clinical research, offering a scalable, efficient alternative to traditional data mining methods without the need for specialized medical language training.
Collapse
Affiliation(s)
- Daniel M Goldenholz
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | - Shira R Goldenholz
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | - Sara Habib
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | - M Brandon Westover
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| |
Collapse
|
15
|
Sushil M, Butte AJ, Schuit E, van Smeden M, Leeuwenberg AM. Cross-institution natural language processing for reliable clinical association studies: a methodological exploration. J Clin Epidemiol 2024; 167:111258. [PMID: 38219811 DOI: 10.1016/j.jclinepi.2024.111258] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 12/21/2023] [Accepted: 01/08/2024] [Indexed: 01/16/2024]
Abstract
OBJECTIVES Natural language processing (NLP) of clinical notes in electronic medical records is increasingly used to extract otherwise sparsely available patient characteristics, to assess their association with relevant health outcomes. Manual data curation is resource intensive and NLP methods make these studies more feasible. However, the methodology of using NLP methods reliably in clinical research is understudied. The objective of this study is to investigate how NLP models could be used to extract study variables (specifically exposures) to reliably conduct exposure-outcome association studies. STUDY DESIGN AND SETTING In a convenience sample of patients admitted to the intensive care unit of a US academic health system, multiple association studies are conducted, comparing the association estimates based on NLP-extracted vs. manually extracted exposure variables. The association studies varied in NLP model architecture (Bidirectional Encoder Decoder from Transformers, Long Short-Term Memory), training paradigm (training a new model, fine-tuning an existing external model), extracted exposures (employment status, living status, and substance use), health outcomes (having a do-not-resuscitate/intubate code, length of stay, and in-hospital mortality), missing data handling (multiple imputation vs. complete case analysis), and the application of measurement error correction (via regression calibration). RESULTS The study was conducted on 1,174 participants (median [interquartile range] age, 61 [50, 73] years; 60.6% male). Additionally, up to 500 discharge reports of participants from the same health system and 2,528 reports of participants from an external health system were used to train the NLP models. Substantial differences were found between the associations based on NLP-extracted and manually extracted exposures under all settings. The error in association was only weakly correlated with the overall F1 score of the NLP models. CONCLUSION Associations estimated using NLP-extracted exposures should be interpreted with caution. Further research is needed to set conditions for reliable use of NLP in medical association studies.
Collapse
Affiliation(s)
- Madhumita Sushil
- Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, USA
| | - Atul J Butte
- Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, USA
| | - Ewoud Schuit
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Artuur M Leeuwenberg
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
| |
Collapse
|
16
|
Büttner M, Leser U, Schneider L, Schwendicke F. Natural Language Processing: Chances and Challenges in Dentistry. J Dent 2024; 141:104796. [PMID: 38072335 DOI: 10.1016/j.jdent.2023.104796] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Revised: 11/25/2023] [Accepted: 11/27/2023] [Indexed: 12/21/2023] Open
Abstract
INTRODUCTION Natural language processing (NLP) is an intersection between Computer Science and Linguistic which aims to enable machines to process and understand human language. We here summarized applications and limitations of NLP in dentistry. DATA AND SOURCES Narrative review. FINDINGS NLP has evolved increasingly fast. For the dental domain, relevant NLP applications are text classification (e.g., symptom classification) and natural language generation and understanding (e.g., clinical chatbots assisting professionals in office work and patient communication). Analyzing large quantities of text will allow understanding diseases and their trajectories and support a more precise and personalized care. Speech recognition systems may serve as virtual assistants and facilitate automated documentation. However, to date, NLP has rarely been applied in dentistry. Existing research focuses mainly on rule-based solutions for narrow tasks. Technologies such as Recurrent Neural Networks and Transformers have been shown to surpass the language processing capabilities of such rule-based solutions in many fields, but are data-hungry (i.e., rely on large amounts of training data), which limits their application in the dental domain at present. Technologies such as federated or transfer learning or data sharing concepts may allow to overcome this limitation, while challenges in terms of explainability, reproducibility, generalizability and evaluation of NLP in dentistry remain to be resolved for enabling approval of such technologies in medical devices and services. CONCLUSIONS NLP will become a cornerstone of a number of applications in dentistry. The community is called to action to improve the current limitations and foster reliable, high-quality dental NLP. CLINICAL SIGNIFICANCE NLP for text classification (e.g., dental symptom classification) and language generation and understanding (e.g., clinical chatbots, speech recognition) will support administrative tasks in dentistry, provide deeper insights for clinicians and support research and education.
Collapse
Affiliation(s)
- Martha Büttner
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Germany.
| | - Ulf Leser
- Department of Computer Science, Humboldt-Universität zu Berlin, Berlin, Germany
| | - Lisa Schneider
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Germany
| | - Falk Schwendicke
- Clinic for Operative, Preventive and Pediatric Dentistry and Periodontology, Ludwig-Maximilians-University, Munich, Germany
| |
Collapse
|
17
|
Feng W, Huang Q, Ma T, Ju L, Ge Z, Chen Y, Zhao P. Development and validation of a semi-supervised deep learning model for automatic retinopathy of prematurity staging. iScience 2024; 27:108516. [PMID: 38269093 PMCID: PMC10805639 DOI: 10.1016/j.isci.2023.108516] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Revised: 07/03/2023] [Accepted: 11/20/2023] [Indexed: 01/26/2024] Open
Abstract
Retinopathy of prematurity (ROP) is currently one of the leading causes of infant blindness worldwide. Recently significant progress has been made in deep learning-based computer-aided diagnostic methods. However, deep learning often requires a large amount of annotated data for model optimization, but this requires long hours of effort by experienced doctors in clinical scenarios. In contrast, a large number of unlabeled images are relatively easy to obtain. In this paper, we propose a new semi-supervised learning framework to reduce annotation costs for automatic ROP staging. We design two consistency regularization strategies, prediction consistency loss and semantic structure consistency loss, which can help the model mine useful discriminative information from unlabeled data, thus improving the generalization performance of the classification model. Extensive experiments on a real clinical dataset show that the proposed method promises to greatly reduce the labeling requirements in clinical scenarios while achieving good classification performance.
Collapse
Affiliation(s)
- Wei Feng
- Beijing Airdoc Technology Co., Ltd, Beijing 100089, China
- Faculty of Engineering, Monash University, Melbourne, VIC 3000, Australia
| | - Qiujing Huang
- Department of Ophthalmology, Xinhua Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai 200092, China
- Department of Ophthalmology, Rainbow Children’s Clinic, Shanghai 200010, China
| | - Tong Ma
- Beijing Airdoc Technology Co., Ltd, Beijing 100089, China
| | - Lie Ju
- Beijing Airdoc Technology Co., Ltd, Beijing 100089, China
- Faculty of Engineering, Monash University, Melbourne, VIC 3000, Australia
| | - Zongyuan Ge
- Faculty of Engineering, Monash University, Melbourne, VIC 3000, Australia
| | - Yuzhong Chen
- Beijing Airdoc Technology Co., Ltd, Beijing 100089, China
| | - Peiquan Zhao
- Department of Ophthalmology, Xinhua Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai 200092, China
| |
Collapse
|
18
|
Zuo X, Zhou Y, Duke J, Hripcsak G, Shah N, Banda JM, Reeves R, Miller T, Waitman LR, Natarajan K, Xu H. Standardizing Multi-site Clinical Note Titles to LOINC Document Ontology: A Transformer-based Approach. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2024; 2023:834-843. [PMID: 38222429 PMCID: PMC10785935] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
The types of clinical notes in electronic health records (EHRs) are diverse and it would be great to standardize them to ensure unified data retrieval, exchange, and integration. The LOINC Document Ontology (DO) is a subset of LOINC that is created specifically for naming and describing clinical documents. Despite the efforts of promoting and improving this ontology, how to efficiently deploy it in real-world clinical settings has yet to be explored. In this study we evaluated the utility of LOINC DO by mapping clinical note titles collected from five institutions to the LOINC DO and classifying the mapping into three classes based on semantic similarity between note titles and LOINC DO codes. Additionally, we developed a standardization pipeline that automatically maps clinical note titles from multiple sites to suitable LOINC DO codes, without accessing the content of clinical notes. The pipeline can be initialized with different large language models, and we compared the performances between them. The results showed that our automated pipeline achieved an accuracy of 0.90. By comparing the manual and automated mapping results, we analyzed the coverage of LOINC DO in describing multi-site clinical note titles and summarized the potential scope for extension.
Collapse
Affiliation(s)
- Xu Zuo
- University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Yujia Zhou
- University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Jon Duke
- Georgia Institute of Technology, Atlanta, GA, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | - George Hripcsak
- Columbia University, New York City, NY, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | - Nigam Shah
- Stanford University, Stanford, CA, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | - Juan M Banda
- Georgia State University, Atlanta, GA, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | - Ruth Reeves
- Vanderbilt University Medical Center, Nashville, TN, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | - Timothy Miller
- Boston Children's Hospital, Boston, MA, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | | | - Karthik Natarajan
- Columbia University, New York City, NY, USA
- OHDSI Consortium, Natural Language Processing Working Group
| | - Hua Xu
- Yale University, New Haven, CT, USA
- OHDSI Consortium, Natural Language Processing Working Group
| |
Collapse
|
19
|
Wu Y, Conlan D, Perez S, Nguyen A. Leveraging Unlabeled Clinical Data to Boost Performance of Risk Stratification Models for Suspected Acute Coronary Syndrome. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2024; 2023:744-753. [PMID: 38222439 PMCID: PMC10785873] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
The performance of deep learning models in the health domain is desperately limited by the scarcity of labeled data, especially for specific clinical-domain tasks. Conversely, there are vastly available clinical unlabeled data waiting to be exploited to improve deep learning models where their training labeled data are limited. This paper investigates the use of task-specific unlabeled data to boost the performance of classification models for the risk stratification of suspected acute coronary syndrome. By leveraging large numbers of unlabeled clinical notes in task-adaptive language model pretraining, valuable prior task-specific knowledge can be attained. Based on such pretrained models, task-specific fine-tuning with limited labeled data produces better performances. Extensive experiments demonstrate that the pretrained task-specific language models using task-specific unlabeled data can significantly improve the performance of the downstream models for specific classification tasks.
Collapse
Affiliation(s)
- Yutong Wu
- The Australian e-Health Research Centre, CSIRO, Brisbane, Queensland, Australia
| | - David Conlan
- The Australian e-Health Research Centre, CSIRO, Brisbane, Queensland, Australia
| | - Siegfried Perez
- Faculty of Medicine, The University of Queensland, Brisbane, Queensland, Australia
- Department of Emergency Medicine, Logan Hospital, Meadowbrook, Queensland, Australia
| | - Anthony Nguyen
- The Australian e-Health Research Centre, CSIRO, Brisbane, Queensland, Australia
| |
Collapse
|
20
|
Pathak A, Yu Z, Paredes D, Monsour EP, Rocha AO, Brito JP, Ospina NS, Wu Y. Extracting Thyroid Nodules Characteristics from Ultrasound Reports Using Transformer-based Natural Language Processing Methods. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2024; 2023:1193-1200. [PMID: 38222394 PMCID: PMC10785862] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
The ultrasound characteristics of thyroid nodules guide the evaluation of thyroid cancer in patients with thyroid nodules. However, the characteristics of thyroid nodules are often documented in clinical narratives such as ultrasound reports. Previous studies have examined natural language processing (NLP) methods in extracting a limited number of characteristics (<9) using rule-based NLP systems. In this study, a multidisciplinary team of NLP experts and thyroid specialists, identified thyroid nodule characteristics that are important for clinical care, composed annotation guidelines, developed a corpus, and compared 5 state-of-the-art transformer-based NLP methods, including BERT, RoBERTa, LongFormer, DeBERTa, and GatorTron, for extraction of thyroid nodule characteristics from ultrasound reports. Our GatorTron model, a transformer-based large language model trained using over 90 billion words of text, achieved the best strict and lenient F1-score of 0.8851 and 0.9495 for the extraction of a total number of 16 thyroid nodule characteristics, and 0.9321 for linking characteristics to nodules, outperforming other clinical transformer models. To the best of our knowledge, this is the first study to systematically categorize and apply transformer-based NLP models to extract a large number of clinical relevant thyroid nodule characteristics from ultrasound reports. This study lays ground for assessing the documentation quality of thyroid ultrasound reports and examining outcomes of patients with thyroid nodules using electronic health records.
Collapse
Affiliation(s)
- Aman Pathak
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Zehao Yu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Daniel Paredes
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Elio Paul Monsour
- Division of Endocrinology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Andrea Ortiz Rocha
- Division of Endocrinology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Juan P Brito
- Division of Endocrinology, Diabetes, Metabolism and Nutrition, Mayo Clinic Rochester, USA
| | - Naykky Singh Ospina
- Division of Endocrinology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Yonghui Wu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| |
Collapse
|
21
|
Ding X, Sheng Z, Yetişgen M, Pakhomov S, Cohen T. Backdoor Adjustment of Confounding by Provenance for Robust Text Classification of Multi-institutional Clinical Notes. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2024; 2023:923-932. [PMID: 38222433 PMCID: PMC10785933] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
Natural Language Processing (NLP) methods have been broadly applied to clinical tasks. Machine learning and deep learning approaches have been used to improve the performance of clinical NLP. However, these approaches require sufficiently large datasets for training, and trained models have been shown to transfer poorly across sites. These issues have led to the promotion of data collection and integration across different institutions for accurate and portable models. However, this can introduce a form of bias called confounding by provenance. When source-specific data distributions differ at deployment, this may harm model performance. To address this issue, we evaluate the utility of backdoor adjustment for text classification in a multi-site dataset of clinical notes annotated for mentions of substance abuse. Using an evaluation framework devised to measure robustness to distributional shifts, we assess the utility of backdoor adjustment. Our results indicate that backdoor adjustment can effectively mitigate for confounding shift.
Collapse
Affiliation(s)
- Xiruo Ding
- University of Washington, Seattle, WA, USA
| | | | | | | | | |
Collapse
|
22
|
Zahra FA, Kate RJ. Obtaining clinical term embeddings from SNOMED CT ontology. J Biomed Inform 2024; 149:104560. [PMID: 38070816 DOI: 10.1016/j.jbi.2023.104560] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 11/29/2023] [Accepted: 12/05/2023] [Indexed: 01/22/2024]
Abstract
Clinical term embeddings are traditionally obtained using corpus-based methods, however, these methods cannot incorporate knowledge about clinical terms which is already present in medical ontologies. On the other hand, graph-based methods can obtain embeddings of clinical concepts from ontologies, but they cannot obtain embeddings for clinical terms and words. In this paper, a novel method is presented to obtain embeddings for clinical terms and words from the SNOMED CT ontology. The method first obtains embeddings of clinical concepts from SNOMED CT using a graph-based method. Next, these concept embeddings are used as targets to train a deep learning model to map clinical terms to concepts embeddings. The learned model then provides embeddings for clinical terms and words as well as maps novel clinical terms to their embeddings. The embeddings obtained using the method out-performed corpus-based embeddings on the task of predicting clinical term similarity on five benchmark datasets. On the clinical term normalization task, using these embeddings simply as a means of computing similarity between clinical terms obtained accuracy which was competitive to methods trained specifically for this task. Both corpus-based and ontology-based embeddings have a limitation that they tend to learn similar embeddings for opposite or analogous terms. To counter this, we also introduce a method to automatically learn patterns that indicate when two clinical terms represent the same concept and when they represent different concepts. Supplementing the normalization process with these patterns showed improvement. Although clinical term embeddings obtained from SNOMED CT incorporate ontological knowledge which is missed by corpus-based embeddings, they do not incorporate linguistic knowledge which is needed for sentence-based tasks. Hence combining ontology-based embeddings with corpus-based embeddings is an avenue for future work.
Collapse
Affiliation(s)
- Fuad Abu Zahra
- Department of Computer Science, University of Wisconsin-Milwaukee, Milwaukee, WI, USA
| | - Rohit J Kate
- Department of Computer Science, University of Wisconsin-Milwaukee, Milwaukee, WI, USA.
| |
Collapse
|
23
|
Swaminathan A, Lopez I, Wang W, Srivastava U, Tran E, Bhargava-Shah A, Wu JY, Ren AL, Caoili K, Bui B, Alkhani L, Lee S, Mohit N, Seo N, Macedo N, Cheng W, Liu C, Thomas R, Chen JH, Gevaert O. Selective prediction for extracting unstructured clinical data. J Am Med Inform Assoc 2023; 31:188-197. [PMID: 37769323 PMCID: PMC10746316 DOI: 10.1093/jamia/ocad182] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Revised: 08/21/2023] [Accepted: 08/24/2023] [Indexed: 09/30/2023] Open
Abstract
OBJECTIVE While there are currently approaches to handle unstructured clinical data, such as manual abstraction and structured proxy variables, these methods may be time-consuming, not scalable, and imprecise. This article aims to determine whether selective prediction, which gives a model the option to abstain from generating a prediction, can improve the accuracy and efficiency of unstructured clinical data abstraction. MATERIALS AND METHODS We trained selective classifiers (logistic regression, random forest, support vector machine) to extract 5 variables from clinical notes: depression (n = 1563), glioblastoma (GBM, n = 659), rectal adenocarcinoma (DRA, n = 601), and abdominoperineal resection (APR, n = 601) and low anterior resection (LAR, n = 601) of adenocarcinoma. We varied the cost of false positives (FP), false negatives (FN), and abstained notes and measured total misclassification cost. RESULTS The depression selective classifiers abstained on anywhere from 0% to 97% of notes, and the change in total misclassification cost ranged from -58% to 9%. Selective classifiers abstained on 5%-43% of notes across the GBM and colorectal cancer models. The GBM selective classifier abstained on 43% of notes, which led to improvements in sensitivity (0.94 to 0.96), specificity (0.79 to 0.96), PPV (0.89 to 0.98), and NPV (0.88 to 0.91) when compared to a non-selective classifier and when compared to structured proxy variables. DISCUSSION We showed that selective classifiers outperformed both non-selective classifiers and structured proxy variables for extracting data from unstructured clinical notes. CONCLUSION Selective prediction should be considered when abstaining is preferable to making an incorrect prediction.
Collapse
Affiliation(s)
- Akshay Swaminathan
- Stanford University School of Medicine, Stanford, CA, United States
- Cerebral Inc. Claymont, DE, United States
| | - Ivan Lopez
- Stanford University School of Medicine, Stanford, CA, United States
- Cerebral Inc. Claymont, DE, United States
| | - William Wang
- Department of Biology, Stanford University, Stanford, CA, United States
- Department of Bioengineering, Stanford University, Stanford, CA, United States
| | - Ujwal Srivastava
- Department of Computer Science, Stanford University, Stanford, CA, United States
| | - Edward Tran
- Department of Computer Science, Stanford University, Stanford, CA, United States
- Department of Management Science and Engineering, Stanford University, Stanford, CA, United States
| | | | - Janet Y Wu
- Stanford University School of Medicine, Stanford, CA, United States
| | - Alexander L Ren
- Stanford University School of Medicine, Stanford, CA, United States
| | - Kaitlin Caoili
- Stanford University School of Medicine, Stanford, CA, United States
| | - Brandon Bui
- Department of Human Biology, Stanford University, Stanford, CA, United States
| | - Layth Alkhani
- Department of Bioengineering, Stanford University, Stanford, CA, United States
- Department of Chemistry, Stanford University, Stanford, CA, United States
| | - Susan Lee
- Department of Computer Science, Stanford University, Stanford, CA, United States
| | - Nathan Mohit
- Department of Computer Science, Stanford University, Stanford, CA, United States
- Department of Human Biology, Stanford University, Stanford, CA, United States
| | - Noel Seo
- Department of Sociology, Stanford University, Stanford, CA, United States
| | - Nicholas Macedo
- Department of Biology, Stanford University, Stanford, CA, United States
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, United States
| | - Winson Cheng
- Department of Computer Science, Stanford University, Stanford, CA, United States
- Department of Chemistry, Stanford University, Stanford, CA, United States
| | - Charles Liu
- Department of Surgery, Stanford University School of Medicine, Stanford, CA, United States
| | - Reena Thomas
- Department of Neurology and Neurological Sciences, Stanford Health Care, Stanford, CA, United States
| | - Jonathan H Chen
- Stanford Center for Biomedical Informatics Research, Stanford, CA, United States
- Division of Hospital Medicine, Stanford, CA, United States
- Clinical Excellence Research Center, Stanford, CA, United States
- Department of Medicine, Stanford, CA, United States
| | - Olivier Gevaert
- Stanford Center for Biomedical Informatics Research, Stanford, CA, United States
- Department of Medicine, Stanford, CA, United States
| |
Collapse
|
24
|
Stewart J, Lu J, Goudie A, Arendts G, Meka SA, Freeman S, Walker K, Sprivulis P, Sanfilippo F, Bennamoun M, Dwivedi G. Applications of natural language processing at emergency department triage: A narrative review. PLoS One 2023; 18:e0279953. [PMID: 38096321 PMCID: PMC10721204 DOI: 10.1371/journal.pone.0279953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2022] [Accepted: 11/30/2023] [Indexed: 12/18/2023] Open
Abstract
INTRODUCTION Natural language processing (NLP) uses various computational methods to analyse and understand human language, and has been applied to data acquired at Emergency Department (ED) triage to predict various outcomes. The objective of this scoping review is to evaluate how NLP has been applied to data acquired at ED triage, assess if NLP based models outperform humans or current risk stratification techniques when predicting outcomes, and assess if incorporating free-text improve predictive performance of models when compared to predictive models that use only structured data. METHODS All English language peer-reviewed research that applied an NLP technique to free-text obtained at ED triage was eligible for inclusion. We excluded studies focusing solely on disease surveillance, and studies that used information obtained after triage. We searched the electronic databases MEDLINE, Embase, Cochrane Database of Systematic Reviews, Web of Science, and Scopus for medical subject headings and text keywords related to NLP and triage. Databases were last searched on 01/01/2022. Risk of bias in studies was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). Due to the high level of heterogeneity between studies and high risk of bias, a metanalysis was not conducted. Instead, a narrative synthesis is provided. RESULTS In total, 3730 studies were screened, and 20 studies were included. The population size varied greatly between studies ranging from 1.8 million patients to 598 triage notes. The most common outcomes assessed were prediction of triage score, prediction of admission, and prediction of critical illness. NLP models achieved high accuracy in predicting need for admission, triage score, critical illness, and mapping free-text chief complaints to structured fields. Incorporating both structured data and free-text data improved results when compared to models that used only structured data. However, the majority of studies (80%) were assessed to have a high risk of bias, and only one study reported the deployment of an NLP model into clinical practice. CONCLUSION Unstructured free-text triage notes have been used by NLP models to predict clinically relevant outcomes. However, the majority of studies have a high risk of bias, most research is retrospective, and there are few examples of implementation into clinical practice. Future work is needed to prospectively assess if applying NLP to data acquired at ED triage improves ED outcomes when compared to usual clinical practice.
Collapse
Affiliation(s)
- Jonathon Stewart
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Juan Lu
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Western Australia, Australia
| | - Adrian Goudie
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Glenn Arendts
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Shiv Akarsh Meka
- HIVE & Data and Digital Innovation, Royal Perth Hospital, Perth, Western Australia, Australia
| | - Sam Freeman
- Department of Emergency Medicine, St Vincent’s Hospital Melbourne, Melbourne, Victoria, Australia
- SensiLab, Monash University, Melbourne, Victoria, Australia
| | - Katie Walker
- School of Clinical Sciences at Monash Health, Monash University, Melbourne, Victoria, Australia
| | - Peter Sprivulis
- Western Australia Department of Health, East Perth, Western Australia, Australia
| | - Frank Sanfilippo
- School of Population and Global Health, University of Western Australia, Crawley, Western Australia, Australia
| | - Mohammed Bennamoun
- Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Western Australia, Australia
| | - Girish Dwivedi
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Cardiology, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| |
Collapse
|
25
|
Lan Z, Turchin A. Impact of possible errors in natural language processing-derived data on downstream epidemiologic analysis. JAMIA Open 2023; 6:ooad111. [PMID: 38152447 PMCID: PMC10752385 DOI: 10.1093/jamiaopen/ooad111] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 12/14/2023] [Accepted: 12/19/2023] [Indexed: 12/29/2023] Open
Abstract
Objective To assess the impact of potential errors in natural language processing (NLP) on the results of epidemiologic studies. Materials and Methods We utilized data from three outcomes research studies where the primary predictor variable was generated using NLP. For each of these studies, Monte Carlo simulations were applied to generate datasets simulating potential errors in NLP-derived variables. We subsequently fit the original regression models to these partially simulated datasets and compared the distribution of coefficient estimates to the original study results. Results Among the four models evaluated, the mean change in the point estimate of the relationship between the predictor variable and the outcome ranged from -21.9% to 4.12%. In three of the four models, significance of this relationship was not eliminated in a single of the 500 simulations, and in one model it was eliminated in 12% of simulations. Mean changes in the estimates for confounder variables ranged from 0.27% to 2.27% and significance of the relationship was eliminated between 0% and 9.25% of the time. No variables underwent a shift in the direction of its interpretation. Discussion Impact of simulated NLP errors on the results of epidemiologic studies was modest, with only small changes in effect estimates and no changes in the interpretation of the findings (direction and significance of association with the outcome) for either the NLP-generated variables or other variables in the models. Conclusion NLP errors are unlikely to affect the results of studies that use NLP as the source of data.
Collapse
Affiliation(s)
- Zhou Lan
- Center for Clinical Investigation, Brigham & Women’s Hospital, Boston, MA 02115, United States
- Harvard Medical School, Boston, MA 02115, United States
| | - Alexander Turchin
- Harvard Medical School, Boston, MA 02115, United States
- Division of Endocrinology, Brigham & Women’s Hospital, Boston, MA 02115, United States
| |
Collapse
|
26
|
Wang M, Yang B, Liu Y, Yang Y, Ji H, Yang C. Emerging infectious disease surveillance using a hierarchical diagnosis model and the Knox algorithm. Sci Rep 2023; 13:19836. [PMID: 37963966 PMCID: PMC10645817 DOI: 10.1038/s41598-023-47010-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Accepted: 11/07/2023] [Indexed: 11/16/2023] Open
Abstract
Emerging infectious diseases are a critical public health challenge in the twenty-first century. The recent proliferation of such diseases has raised major social and economic concerns. Therefore, early detection of emerging infectious diseases is essential. Subjects from five medical institutions in Beijing, China, which met the spatial-specific requirements, were analyzed. A quality control process was used to select 37,422 medical records of infectious diseases and 56,133 cases of non-infectious diseases. An emerging infectious disease detection model (EIDDM), a two-layer model that divides the problem into two sub-problems, i.e., whether a case is an infectious disease, and if so, whether it is a known infectious disease, was proposed. The first layer model adopts the binary classification model TextCNN-Attention. The second layer is a multi-classification model of LightGBM based on the one-vs-rest strategy. Based on the experimental results, a threshold of 0.5 is selected. The model results were compared with those of other models such as XGBoost and Random Forest using the following evaluation indicators: accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. The prediction performance of the first-layer TextCNN is better than that of other comparison models. Its average specificity for non-infectious diseases is 97.57%, with an average negative predictive value of 82.63%, indicating a low risk of misdiagnosing non-infectious diseases as infectious (i.e., a low false positive rate). Its average positive predictive value for eight selected infectious diseases is 95.07%, demonstrating the model's ability to avoid misdiagnoses. The overall average accuracy of the model is 86.11%. The average prediction accuracy of the second-layer LightGBM model for emerging infectious diseases reaches 90.44%. Furthermore, the response time of a single online reasoning using the LightGBM model is approximately 27 ms, which makes it suitable for analyzing clinical records in real time. Using the Knox method, we found that all the infectious diseases were within 2000 m in our case, and a clustering feature of spatiotemporal interactions (P < 0.05) was observed as well. Performance testing and model comparison results indicated that the EIDDM is fast and accurate and can be used to monitor the onset/outbreak of emerging infectious diseases in real-world hospitals.
Collapse
Affiliation(s)
- Mengying Wang
- State Key Laboratory of Media Convergence and Communication, Communication University of China, No. 1, Dingfuzhuang East Street, Chaoyang District, Beijing, China
- Information Management and Big Data Center, Peking University Third Hospital, No. 49, Huayuan North Road, Beijing, China
| | - Bingqing Yang
- Goodwill Hessian Health Technology Co. Ltd, Beijing, China
| | - Yunpeng Liu
- Goodwill Hessian Health Technology Co. Ltd, Beijing, China
| | - Yingyun Yang
- State Key Laboratory of Media Convergence and Communication, Communication University of China, No. 1, Dingfuzhuang East Street, Chaoyang District, Beijing, China
| | - Hong Ji
- Information Management and Big Data Center, Peking University Third Hospital, No. 49, Huayuan North Road, Beijing, China.
| | - Cheng Yang
- State Key Laboratory of Media Convergence and Communication, Communication University of China, No. 1, Dingfuzhuang East Street, Chaoyang District, Beijing, China.
| |
Collapse
|
27
|
Lin Y, Li J, Xiao H, Zheng L, Xiao Y, Song H, Fan J, Xiao D, Ai D, Fu T, Wang F, Lv H, Yang J. Automatic literature screening using the PAJO deep-learning model for clinical practice guidelines. BMC Med Inform Decis Mak 2023; 23:247. [PMID: 37924054 PMCID: PMC10625217 DOI: 10.1186/s12911-023-02328-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2023] [Accepted: 10/06/2023] [Indexed: 11/06/2023] Open
Abstract
BACKGROUND Clinical practice guidelines (CPGs) are designed to assist doctors in clinical decision making. High-quality research articles are important for the development of good CPGs. Commonly used manual screening processes are time-consuming and labor-intensive. Artificial intelligence (AI)-based techniques have been widely used to analyze unstructured data, including texts and images. Currently, there are no effective/efficient AI-based systems for screening literature. Therefore, developing an effective method for automatic literature screening can provide significant advantages. METHODS Using advanced AI techniques, we propose the Paper title, Abstract, and Journal (PAJO) model, which treats article screening as a classification problem. For training, articles appearing in the current CPGs are treated as positive samples. The others are treated as negative samples. Then, the features of the texts (e.g., titles and abstracts) and journal characteristics are fully utilized by the PAJO model using the pretrained bidirectional-encoder-representations-from-transformers (BERT) model. The resulting text and journal encoders, along with the attention mechanism, are integrated in the PAJO model to complete the task. RESULTS We collected 89,940 articles from PubMed to construct a dataset related to neck pain. Extensive experiments show that the PAJO model surpasses the state-of-the-art baseline by 1.91% (F1 score) and 2.25% (area under the receiver operating characteristic curve). Its prediction performance was also evaluated with respect to subject-matter experts, proving that PAJO can successfully screen high-quality articles. CONCLUSIONS The PAJO model provides an effective solution for automatic literature screening. It can screen high-quality articles on neck pain and significantly improve the efficiency of CPG development. The methodology of PAJO can also be easily extended to other diseases for literature screening.
Collapse
Affiliation(s)
- Yucong Lin
- School of Medical Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Jia Li
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, Beijing, 100050, China
| | - Huan Xiao
- School of Statistics, Renmin University of China, Beijing, 100872, China
| | - Lujie Zheng
- School of Computer Science & Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Ying Xiao
- School of Automation, Beijing Institute of Technology, Beijing, 100081, China
| | - Hong Song
- School of Computer Science & Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Jingfan Fan
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Deqiang Xiao
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Danni Ai
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Tianyu Fu
- School of Medical Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Feifei Wang
- School of Statistics, Renmin University of China, Beijing, 100872, China.
- Center for Applied Statistics, Renmin University of China, Beijing, 100872, China.
| | - Han Lv
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, Beijing, 100050, China.
| | - Jian Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China.
| |
Collapse
|
28
|
Hua Y, Wang L, Nguyen V, Rieu-Werden M, McDowell A, Bates DW, Foer D, Zhou L. A deep learning approach for transgender and gender diverse patient identification in electronic health records. J Biomed Inform 2023; 147:104507. [PMID: 37778672 PMCID: PMC10687838 DOI: 10.1016/j.jbi.2023.104507] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 09/18/2023] [Accepted: 09/22/2023] [Indexed: 10/03/2023]
Abstract
BACKGROUND Although accurate identification of gender identity in the electronic health record (EHR) is crucial for providing equitable health care, particularly for transgender and gender diverse (TGD) populations, it remains a challenging task due to incomplete gender information in structured EHR fields. OBJECTIVE Using TGD identification as a case study, this research uses NLP and deep learning to build an accurate patient gender identity predictive model, aiming to tackle the challenges of identifying relevant patient-level information from EHR data and reducing annotation work. METHODS This study included adult patients in a large healthcare system in Boston, MA, between 4/1/2017 to 4/1/2022. To identify relevant information from massive clinical notes, we compiled a list of gender-related keywords through expert curation, literature review, and expansion via a fine-tuned BioWordVec model. This keyword list was used to pre-screen potential TGD individuals and create two datasets for model training, testing, and validation. Dataset I was a balanced dataset that contained clinician-confirmed TGD patients and cases without keywords. Dataset II contained cases with keywords. The performance of the deep learning model was compared to traditional machine learning and rule-based algorithms. RESULTS The final keyword list consists of 109 keywords, of which 58 (53.2%) were expanded by the BioWordVec model. Dataset I contained 3,150 patients (50% TGD) while Dataset II contained 200 patients (90% TGD). On Dataset I the deep learning model achieved a F1 score of 0.917, sensitivity of 0.854, and a precision of 0.980; and on Dataset II a F1 score of 0.969, sensitivity of 0.967, and precision of 0.972. The deep learning model significantly outperformed rule-based algorithms. CONCLUSION This is the first study to show that deep learning-integrated NLP algorithms can accurately identify gender identity using EHR data. Future work should leverage and evaluate additional diverse data sources to generate more generalizable algorithms.
Collapse
Affiliation(s)
- Yining Hua
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA; Department of Epidemiology, Harvard T.H Chan School of Public Health, Boston, MA, USA; Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
| | - Liqin Wang
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA.
| | - Vi Nguyen
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA.
| | - Meghan Rieu-Werden
- Division of General Medicine, Massachusetts General Hospital, Boston, MA, USA.
| | - Alex McDowell
- Health Policy Research Institute, Mongan Institute, Massachusetts General Hospital, Boston, MA, USA; Department of Health Care Policy, Harvard Medical School, Boston, MA, USA.
| | - David W Bates
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA.
| | - Dinah Foer
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA; Division of Allergy and Clinical Immunology, Department of Medicine, Brigham and Women's Hospital, USA.
| | - Li Zhou
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
29
|
Perez N, Cuadros M, Rigau G. Negation and speculation processing: A study on cue-scope labelling and assertion classification in Spanish clinical text. Artif Intell Med 2023; 145:102682. [PMID: 37925211 DOI: 10.1016/j.artmed.2023.102682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Revised: 08/25/2023] [Accepted: 10/06/2023] [Indexed: 11/06/2023]
Abstract
Natural Language Processing (NLP) based on new deep learning technology is contributing to the emergence of powerful solutions that help healthcare providers and researchers discover valuable patterns within insurmountable volumes of health records and scientific literature. Fundamental to the success of such solutions is the processing of negation and speculation. The article addresses this problem with state-of-the-art deep learning approaches from two perspectives: cue and scope labelling, and assertion classification. In light of the real struggle to access clinical annotated data, the study (a) proposes a methodology to automatically convert cue-scope annotations to assertion annotations; and (b) includes a range of scenarios with varying amounts of training data and adversarial test examples. The results expose the clear advantage of Transformer-based models in this regard, managing to overpass a series of baselines and the related work in the public corpus NUBes of clinical Spanish text.
Collapse
Affiliation(s)
- Naiara Perez
- SNLT group at Vicomtech Foundation, Basque Research and Technology Alliance (BRTA), Mikeletegi Pasealekua 57, Donostia/San Sebastián, 20009, Spain; HiTZ Basque Center for Language Technologies, University of the Basque Country (UPV-EHU), Manuel Lardizabal Ibilbidea 1, Donostia/San Sebastián, 20018, Spain.
| | - Montse Cuadros
- SNLT group at Vicomtech Foundation, Basque Research and Technology Alliance (BRTA), Mikeletegi Pasealekua 57, Donostia/San Sebastián, 20009, Spain
| | - German Rigau
- HiTZ Basque Center for Language Technologies, University of the Basque Country (UPV-EHU), Manuel Lardizabal Ibilbidea 1, Donostia/San Sebastián, 20018, Spain
| |
Collapse
|
30
|
Cui R, Wang L, Lin L, Li J, Lu R, Liu S, Liu B, Gu Y, Zhang H, Shang Q, Chen L, Tian D. Deep Learning in Barrett's Esophagus Diagnosis: Current Status and Future Directions. Bioengineering (Basel) 2023; 10:1239. [PMID: 38002363 PMCID: PMC10669008 DOI: 10.3390/bioengineering10111239] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Revised: 10/13/2023] [Accepted: 10/16/2023] [Indexed: 11/26/2023] Open
Abstract
Barrett's esophagus (BE) represents a pre-malignant condition characterized by abnormal cellular proliferation in the distal esophagus. A timely and accurate diagnosis of BE is imperative to prevent its progression to esophageal adenocarcinoma, a malignancy associated with a significantly reduced survival rate. In this digital age, deep learning (DL) has emerged as a powerful tool for medical image analysis and diagnostic applications, showcasing vast potential across various medical disciplines. In this comprehensive review, we meticulously assess 33 primary studies employing varied DL techniques, predominantly featuring convolutional neural networks (CNNs), for the diagnosis and understanding of BE. Our primary focus revolves around evaluating the current applications of DL in BE diagnosis, encompassing tasks such as image segmentation and classification, as well as their potential impact and implications in real-world clinical settings. While the applications of DL in BE diagnosis exhibit promising results, they are not without challenges, such as dataset issues and the "black box" nature of models. We discuss these challenges in the concluding section. Essentially, while DL holds tremendous potential to revolutionize BE diagnosis, addressing these challenges is paramount to harnessing its full capacity and ensuring its widespread application in clinical practice.
Collapse
Affiliation(s)
- Ruichen Cui
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Lei Wang
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
- West China School of Nursing, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China
| | - Lin Lin
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
- West China School of Nursing, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China
| | - Jie Li
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
- West China School of Nursing, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China
| | - Runda Lu
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Shixiang Liu
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Bowei Liu
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Yimin Gu
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Hanlu Zhang
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Qixin Shang
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Longqi Chen
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| | - Dong Tian
- Department of Thoracic Surgery, West China Hospital, Sichuan University, 37 Guoxue Alley, Chengdu 610041, China; (R.C.); (L.W.); (L.L.); (J.L.); (R.L.); (S.L.); (B.L.); (Y.G.); (H.Z.); (Q.S.)
| |
Collapse
|
31
|
Nishioka S, Asano M, Yada S, Aramaki E, Yajima H, Yanagisawa Y, Sayama K, Kizaki H, Hori S. Adverse event signal extraction from cancer patients' narratives focusing on impact on their daily-life activities. Sci Rep 2023; 13:15516. [PMID: 37726371 PMCID: PMC10509234 DOI: 10.1038/s41598-023-42496-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2023] [Accepted: 09/11/2023] [Indexed: 09/21/2023] Open
Abstract
Adverse event (AE) management is important to improve anti-cancer treatment outcomes, but it is known that some AE signals can be missed during clinical visits. In particular, AEs that affect patients' activities of daily living (ADL) need careful monitoring as they may require immediate medical intervention. This study aimed to build deep-learning (DL) models for extracting signals of AEs limiting ADL from patients' narratives. The data source was blog posts written in Japanese by breast cancer patients. After pre-processing and annotation for AE signals, three DL models (BERT, ELECTRA, and T5) were trained and tested in three different approaches for AE signal identification. The performances of the trained models were evaluated in terms of precision, recall, and F1 scores. From 2,272 blog posts, 191 and 702 articles were identified as describing AEs limiting ADL or not limiting ADL, respectively. Among tested DL modes and approaches, T5 showed the best F1 scores to identify articles with AE limiting ADL or all AE: 0.557 and 0.811, respectively. The most frequent AE signals were "pain or numbness", "fatigue" and "nausea". Our results suggest that this AE monitoring scheme focusing on patients' ADL has potential to reinforce current AE management provided by medical staff.
Collapse
Affiliation(s)
- Satoshi Nishioka
- Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan
| | - Masaki Asano
- Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan
| | - Shuntaro Yada
- Nara Institute of Science and Technology, Nara, Japan
| | - Eiji Aramaki
- Nara Institute of Science and Technology, Nara, Japan
| | | | - Yuki Yanagisawa
- Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan
| | - Kyoko Sayama
- Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan
| | - Hayato Kizaki
- Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan
| | - Satoko Hori
- Division of Drug Informatics, Keio University Faculty of Pharmacy, Tokyo, Japan.
| |
Collapse
|
32
|
Zhang H, Ogasawara K. Grad-CAM-Based Explainable Artificial Intelligence Related to Medical Text Processing. Bioengineering (Basel) 2023; 10:1070. [PMID: 37760173 PMCID: PMC10525184 DOI: 10.3390/bioengineering10091070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 08/28/2023] [Accepted: 09/06/2023] [Indexed: 09/29/2023] Open
Abstract
The opacity of deep learning makes its application challenging in the medical field. Therefore, there is a need to enable explainable artificial intelligence (XAI) in the medical field to ensure that models and their results can be explained in a manner that humans can understand. This study uses a high-accuracy computer vision algorithm model to transfer learning to medical text tasks and uses the explanatory visualization method known as gradient-weighted class activation mapping (Grad-CAM) to generate heat maps to ensure that the basis for decision-making can be provided intuitively or via the model. The system comprises four modules: pre-processing, word embedding, classifier, and visualization. We used Word2Vec and BERT to compare word embeddings and use ResNet and 1Dimension convolutional neural networks (CNN) to compare classifiers. Finally, the Bi-LSTM was used to perform text classification for direct comparison. With 25 epochs, the model that used pre-trained ResNet on the formalized text presented the best performance (recall of 90.9%, precision of 91.1%, and an F1 score of 90.2% weighted). This study uses ResNet to process medical texts through Grad-CAM-based explainable artificial intelligence and obtains a high-accuracy classification effect; at the same time, through Grad-CAM visualization, it intuitively shows the words to which the model pays attention when making predictions.
Collapse
Affiliation(s)
| | - Katsuhiko Ogasawara
- Graduate School of Health Science, Hokkaido University, N12-W5, Kitaku, Sapporo 060-0812, Japan
| |
Collapse
|
33
|
Pan J, Zhang Z, Peters SR, Vatanpour S, Walker RL, Lee S, Martin EA, Quan H. Cerebrovascular disease case identification in inpatient electronic medical record data using natural language processing. Brain Inform 2023; 10:22. [PMID: 37658963 PMCID: PMC10474977 DOI: 10.1186/s40708-023-00203-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 08/14/2023] [Indexed: 09/05/2023] Open
Abstract
BACKGROUND Abstracting cerebrovascular disease (CeVD) from inpatient electronic medical records (EMRs) through natural language processing (NLP) is pivotal for automated disease surveillance and improving patient outcomes. Existing methods rely on coders' abstraction, which has time delays and under-coding issues. This study sought to develop an NLP-based method to detect CeVD using EMR clinical notes. METHODS CeVD status was confirmed through a chart review on randomly selected hospitalized patients who were 18 years or older and discharged from 3 hospitals in Calgary, Alberta, Canada, between January 1 and June 30, 2015. These patients' chart data were linked to administrative discharge abstract database (DAD) and Sunrise™ Clinical Manager (SCM) EMR database records by Personal Health Number (a unique lifetime identifier) and admission date. We trained multiple natural language processing (NLP) predictive models by combining two clinical concept extraction methods and two supervised machine learning (ML) methods: random forest and XGBoost. Using chart review as the reference standard, we compared the model performances with those of the commonly applied International Classification of Diseases (ICD-10-CA) codes, on the metrics of sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). RESULT Of the study sample (n = 3036), the prevalence of CeVD was 11.8% (n = 360); the median patient age was 63; and females accounted for 50.3% (n = 1528) based on chart data. Among 49 extracted clinical documents from the EMR, four document types were identified as the most influential text sources for identifying CeVD disease ("nursing transfer report," "discharge summary," "nursing notes," and "inpatient consultation."). The best performing NLP model was XGBoost, combining the Unified Medical Language System concepts extracted by cTAKES (e.g., top-ranked concepts, "Cerebrovascular accident" and "Transient ischemic attack"), and the term frequency-inverse document frequency vectorizer. Compared with ICD codes, the model achieved higher validity overall, such as sensitivity (25.0% vs 70.0%), specificity (99.3% vs 99.1%), PPV (82.6 vs. 87.8%), and NPV (90.8% vs 97.1%). CONCLUSION The NLP algorithm developed in this study performed better than the ICD code algorithm in detecting CeVD. The NLP models could result in an automated EMR tool for identifying CeVD cases and be applied for future studies such as surveillance, and longitudinal studies.
Collapse
Affiliation(s)
- Jie Pan
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.
| | - Zilong Zhang
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Steven Ray Peters
- Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Shabnam Vatanpour
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Robin L Walker
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Alberta Health Services, Edmonton, AB, Canada
| | - Seungwon Lee
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Alberta Health Services, Edmonton, AB, Canada
| | - Elliot A Martin
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Alberta Health Services, Edmonton, AB, Canada
| | - Hude Quan
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| |
Collapse
|
34
|
Fraile Navarro D, Ijaz K, Rezazadegan D, Rahimi-Ardabili H, Dras M, Coiera E, Berkovsky S. Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. Int J Med Inform 2023; 177:105122. [PMID: 37295138 DOI: 10.1016/j.ijmedinf.2023.105122] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2022] [Revised: 04/14/2023] [Accepted: 06/03/2023] [Indexed: 06/12/2023]
Abstract
BACKGROUND Natural Language Processing (NLP) applications have developed over the past years in various fields including its application to clinical free text for named entity recognition and relation extraction. However, there has been rapid developments the last few years that there's currently no overview of it. Moreover, it is unclear how these models and tools have been translated into clinical practice. We aim to synthesize and review these developments. METHODS We reviewed literature from 2010 to date, searching PubMed, Scopus, the Association of Computational Linguistics (ACL), and Association of Computer Machinery (ACM) libraries for studies of NLP systems performing general-purpose (i.e., not disease- or treatment-specific) information extraction and relation extraction tasks in unstructured clinical text (e.g., discharge summaries). RESULTS We included in the review 94 studies with 30 studies published in the last three years. Machine learning methods were used in 68 studies, rule-based in 5 studies, and both in 22 studies. 63 studies focused on Named Entity Recognition, 13 on Relation Extraction and 18 performed both. The most frequently extracted entities were "problem", "test" and "treatment". 72 studies used public datasets and 22 studies used proprietary datasets alone. Only 14 studies defined clearly a clinical or information task to be addressed by the system and just three studies reported its use outside the experimental setting. Only 7 studies shared a pre-trained model and only 8 an available software tool. DISCUSSION Machine learning-based methods have dominated the NLP field on information extraction tasks. More recently, Transformer-based language models are taking the lead and showing the strongest performance. However, these developments are mostly based on a few datasets and generic annotations, with very few real-world use cases. This may raise questions about the generalizability of findings, translation into practice and highlights the need for robust clinical evaluation.
Collapse
Affiliation(s)
- David Fraile Navarro
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia.
| | - Kiran Ijaz
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Dana Rezazadegan
- Department of Computer Science and Software Engineering. School of Software and Electrical Engineering, Swinburne University of Technology, Melbourne, Australia
| | - Hania Rahimi-Ardabili
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Mark Dras
- Department of Computing, Macquarie University, Sydney, Australia
| | - Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Shlomo Berkovsky
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| |
Collapse
|
35
|
Zhao H, Ni P, Zhao Q, Liang X, Ai D, Erhardt S, Wang J, Li Y, Wang J. Identifying the serious clinical outcomes of adverse reactions to drugs by a multi-task deep learning framework. Commun Biol 2023; 6:870. [PMID: 37620651 PMCID: PMC10449791 DOI: 10.1038/s42003-023-05243-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 08/14/2023] [Indexed: 08/26/2023] Open
Abstract
Adverse Drug Reactions (ADRs) have a direct impact on human health. As continuous pharmacovigilance and drug monitoring prove to be costly and time-consuming, computational methods have emerged as promising alternatives. However, most existing computational methods primarily focus on predicting whether or not the drug is associated with an adverse reaction and do not consider the core issue of drug benefit-risk assessment-whether the treatment outcome is serious when adverse drug reactions occur. To this end, we categorize serious clinical outcomes caused by adverse reactions to drugs into seven distinct classes and present a deep learning framework, so-called GCAP, for predicting the seriousness of clinical outcomes of adverse reactions to drugs. GCAP has two tasks: one is to predict whether adverse reactions to drugs cause serious clinical outcomes, and the other is to infer the corresponding classes of serious clinical outcomes. Experimental results demonstrate that our method is a powerful and robust framework with high extendibility. GCAP can serve as a useful tool to successfully address the challenge of predicting the seriousness of clinical outcomes stemming from adverse reactions to drugs.
Collapse
Affiliation(s)
- Haochen Zhao
- School of Computer Science and Engineering, Central South University, Changsha, 410083, China
- Hunan Provincial Key Lab on Bioinformatics, Central South University, Changsha, 410083, China
- Xiangjiang Laboratory, Changsha, 410205, China
| | - Peng Ni
- School of Computer Science and Engineering, Central South University, Changsha, 410083, China
- Hunan Provincial Key Lab on Bioinformatics, Central South University, Changsha, 410083, China
| | - Qichang Zhao
- School of Computer Science and Engineering, Central South University, Changsha, 410083, China
- Hunan Provincial Key Lab on Bioinformatics, Central South University, Changsha, 410083, China
| | - Xiao Liang
- School of Computer Science and Engineering, Central South University, Changsha, 410083, China
- Hunan Provincial Key Lab on Bioinformatics, Central South University, Changsha, 410083, China
| | - Di Ai
- Department of Pathology and Laboratory Medicine, McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, 77030, USA
| | - Shannon Erhardt
- Department of Pediatrics, McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, 77030, USA
| | - Jun Wang
- Department of Pediatrics, McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, 77030, USA
| | - Yaohang Li
- Department of Computer Science, Old Dominion University, Norfolk, VA, 23529-0001, USA
| | - Jianxin Wang
- School of Computer Science and Engineering, Central South University, Changsha, 410083, China.
- Hunan Provincial Key Lab on Bioinformatics, Central South University, Changsha, 410083, China.
- Xiangjiang Laboratory, Changsha, 410205, China.
| |
Collapse
|
36
|
Doi K, Takegawa H, Yui M, Anetai Y, Koike Y, Nakamura S, Tanigawa N, Koziumi M, Nishio T. Deep learning-based detection of patients with bone metastasis from Japanese radiology reports. Jpn J Radiol 2023; 41:900-908. [PMID: 36988827 DOI: 10.1007/s11604-023-01413-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Accepted: 03/07/2023] [Indexed: 03/30/2023]
Abstract
PURPOSE Deep learning (DL) is a state-of-the-art technique for developing artificial intelligence in various domains and it improves the performance of natural language processing (NLP). Therefore, we aimed to develop a DL-based NLP model that classifies the status of bone metastasis (BM) in radiology reports to detect patients with BM. MATERIALS AND METHODS The DL-based NLP model was developed by training long short-term memory using 1,749 free-text radiology reports written in Japanese. We adopted five-fold cross-validation and used 200 reports for testing the five models. The accuracy, sensitivity, specificity, precision, and area under the receiver operating characteristics curve (AUROC) were used for the model evaluation. RESULTS The developed model demonstrated classification performance with mean ± standard deviation of 0.912 ± 0.012, 0.924 ± 0.029, 0.901 ± 0.014, 0.898 ± 0.012, and 0.968 ± 0.004 for accuracy, sensitivity, specificity, precision, and AUROC, respectively. CONCLUSION The proposed DL-based NLP model may help in the early and efficient detection of patients with BM.
Collapse
Affiliation(s)
- Kentaro Doi
- Department of Medical Physics and Engineering, Osaka University Graduate School of Medicine, 1-7 Yamadaoka, Suita-Shi, Osaka, Japan
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
| | - Hideki Takegawa
- Department of Medical Physics and Engineering, Osaka University Graduate School of Medicine, 1-7 Yamadaoka, Suita-Shi, Osaka, Japan.
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan.
- Department of Radiation Oncology, Kansai Medical University Hospital, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan.
| | - Midori Yui
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
- Department of Radiation Oncology, Kansai Medical University Hospital, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
| | - Yusuke Anetai
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
- Department of Radiation Oncology, Kansai Medical University Hospital, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
| | - Yuhei Koike
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
- Department of Radiation Oncology, Kansai Medical University Hospital, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
| | - Satoaki Nakamura
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
- Department of Radiation Oncology, Kansai Medical University Hospital, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
| | - Noboru Tanigawa
- Department of Radiology, Kansai Medical University Graduate School of Medicine, 2-5-1 Shinmachi, Hirakata-Shi, Osaka, Japan
| | - Masahiko Koziumi
- Department of Medical Physics and Engineering, Osaka University Graduate School of Medicine, 1-7 Yamadaoka, Suita-Shi, Osaka, Japan
| | - Teiji Nishio
- Department of Medical Physics and Engineering, Osaka University Graduate School of Medicine, 1-7 Yamadaoka, Suita-Shi, Osaka, Japan
| |
Collapse
|
37
|
Ma T, Wu Q, Jiang L, Zeng X, Wang Y, Yuan Y, Wang B, Zhang T. Artificial Intelligence and Machine (Deep) Learning in Otorhinolaryngology: A Bibliometric Analysis Based on VOSviewer and CiteSpace. EAR, NOSE & THROAT JOURNAL 2023:1455613231185074. [PMID: 37515527 DOI: 10.1177/01455613231185074] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/31/2023] Open
Abstract
BACKGROUND Otorhinolaryngology diseases are well suited for artificial intelligence (AI)-based interpretation. The use of AI, particularly AI based on deep learning (DL), in the treatment of human diseases is becoming more and more popular. However, there are few bibliometric analyses that have systematically studied this field. OBJECTIVE The objective of this study was to visualize the research hot spots and trends of AI and DL in ENT diseases through bibliometric analysis to help researchers understand the future development of basic and clinical research. METHODS In all, 232 articles and reviews were retrieved from The Web of Science Core Collection. Using CiteSpace and VOSviewer software, countries, institutions, authors, references, and keywords in the field were visualized and examined. RESULTS The majority of these papers came from 44 nations and 498 institutions, with China and the United States leading the way. Common diseases used by AI in ENT include otosclerosis, otitis media, nasal polyps, sinusitis, and so on. In the early years, research focused on the analysis of hearing and articulation disorders, and in recent years mainly on the diagnosis, localization, and grading of diseases. CONCLUSIONS The analysis shows the periodical hot spots and development direction of AI and DL application in ENT diseases from the time dimension. The diagnosis and prognosis of otolaryngology diseases and the analysis of otolaryngology endoscopic images have been the focus of current research and the development trend of future.
Collapse
Affiliation(s)
- Tianyu Ma
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Qilong Wu
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Li Jiang
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Xiaoyun Zeng
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Yuyao Wang
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Yi Yuan
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Bingxuan Wang
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Tianhong Zhang
- Department of Otorhinolaryngology Head and Neck Surgery, The First Affiliated Hospital of Harbin Medical University, Harbin, China
| |
Collapse
|
38
|
Crowson MG, Alsentzer E, Fiskio J, Bates DW. Towards Medical Billing Automation: NLP for Outpatient Clinician Note Classification. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.07.07.23292367. [PMID: 37502975 PMCID: PMC10370228 DOI: 10.1101/2023.07.07.23292367] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/29/2023]
Abstract
Objectives Our primary objective was to develop a natural language processing approach that accurately predicts outpatient Evaluation and Management (E/M) level of service (LoS) codes using clinicians' notes from a health system electronic health record. A secondary objective was to investigate the impact of clinic note de-identification on document classification performance. Methods We used retrospective outpatient office clinic notes from four medical and surgical specialties. Classification models were fine-tuned on the clinic notes datasets and stratified by subspecialty. The success criteria for the classification tasks were the classification accuracy and F1-scores on internal test data. For the secondary objective, the dataset was de-identified using Named Entity Recognition (NER) to remove protected health information (PHI), and models were retrained. Results The models demonstrated similar predictive performance across different specialties, except for internal medicine, which had the lowest classification accuracy across all model architectures. The models trained on the entire note corpus achieved an E/M LoS CPT code classification accuracy of 74.8% (CI 95: 74.1-75.6). However, the de-identified note corpus showed a markedly lower classification accuracy of 48.2% (CI 95: 47.7-48.6) compared to the model trained on the identified notes. Conclusion The study demonstrates the potential of NLP-based document classifiers to accurately predict E/M LoS CPT codes using clinical notes from various medical and procedural specialties. The models' performance suggests that the classification task's complexity merits further investigation. The de-identification experiment demonstrated that de-identification may negatively impact classifier performance. Further research is needed to validate the performance of our NLP classifiers in different healthcare settings and patient populations and to investigate the potential implications of de-identification on model performance.
Collapse
|
39
|
Cai L, Li J, Lv H, Liu W, Niu H, Wang Z. Integrating domain knowledge for biomedical text analysis into deep learning: A survey. J Biomed Inform 2023; 143:104418. [PMID: 37290540 DOI: 10.1016/j.jbi.2023.104418] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 04/24/2023] [Accepted: 05/31/2023] [Indexed: 06/10/2023]
Abstract
The past decade has witnessed an explosion of textual information in the biomedical field. Biomedical texts provide a basis for healthcare delivery, knowledge discovery, and decision-making. Over the same period, deep learning has achieved remarkable performance in biomedical natural language processing, however, its development has been limited by well-annotated datasets and interpretability. To solve this, researchers have considered combining domain knowledge (such as biomedical knowledge graph) with biomedical data, which has become a promising means of introducing more information into biomedical datasets and following evidence-based medicine. This paper comprehensively reviews more than 150 recent literature studies on incorporating domain knowledge into deep learning models to facilitate typical biomedical text analysis tasks, including information extraction, text classification, and text generation. We eventually discuss various challenges and future directions.
Collapse
Affiliation(s)
- Linkun Cai
- School of Biological Science and Medical Engineering, Beihang University, 100191 Beijing, China
| | - Jia Li
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, 100050 Beijing, China
| | - Han Lv
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, 100050 Beijing, China
| | - Wenjuan Liu
- Aerospace Center Hospital, 100049 Beijing, China
| | - Haijun Niu
- School of Biological Science and Medical Engineering, Beihang University, 100191 Beijing, China
| | - Zhenchang Wang
- School of Biological Science and Medical Engineering, Beihang University, 100191 Beijing, China; Department of Radiology, Beijing Friendship Hospital, Capital Medical University, 100050 Beijing, China.
| |
Collapse
|
40
|
Schäfer H, Idrissi-Yaghir A, Bewersdorff J, Frihat S, Friedrich CM, Zesch T. Medication event extraction in clinical notes: Contribution of the WisPerMed team to the n2c2 2022 challenge. J Biomed Inform 2023; 143:104400. [PMID: 37211196 DOI: 10.1016/j.jbi.2023.104400] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Revised: 04/21/2023] [Accepted: 05/15/2023] [Indexed: 05/23/2023]
Abstract
In this work, we describe the findings of the 'WisPerMed' team from their participation in Track 1 (Contextualized Medication Event Extraction) of the n2c2 2022 challenge. We tackle two tasks: (i) medication extraction, which involves extracting all mentions of medications from the clinical notes, and (ii) event classification, which involves classifying the medication mentions based on whether a change in the medication has been discussed. To address the long lengths of clinical texts, which often exceed the maximum token length that models based on the transformer-architecture can handle, various approaches, such as the use of ClinicalBERT with a sliding window approach and Longformer-based models, are employed. In addition, domain adaptation through masked language modeling and preprocessing steps such as sentence splitting are utilized to improve model performance. Since both tasks were treated as named entity recognition (NER) problems, a sanity check was performed in the second release to eliminate possible weaknesses in the medication detection itself. This check used the medication spans to remove false positive predictions and replace missed tokens with the highest softmax probability of the disposition types. The effectiveness of these approaches is evaluated through multiple submissions to the tasks, as well as with post-challenge results, with a focus on the DeBERTa v3 model and its disentangled attention mechanism. Results show that the DeBERTa v3 model performs well in both the NER task and the event classification task.
Collapse
Affiliation(s)
- Henning Schäfer
- Department of Computer Science, University of Applied Sciences and Arts Dortmund (FH Dortmund), Emil-Figge-Straße 42, Dortmund, 44227, Germany; Institute for Transfusion Medicine, University Hospital Essen, Hufelandstraße 55, Essen, 45147, Germany.
| | - Ahmad Idrissi-Yaghir
- Department of Computer Science, University of Applied Sciences and Arts Dortmund (FH Dortmund), Emil-Figge-Straße 42, Dortmund, 44227, Germany; Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Hufelandstraße 55, Essen, 45147, Germany
| | - Jeanette Bewersdorff
- Computational Linguistics, CATALPA - Center for Advanced Technology-Assisted Learning and Predictive Analytics, FernUniversität in Hagen, Germany
| | | | - Christoph M Friedrich
- Department of Computer Science, University of Applied Sciences and Arts Dortmund (FH Dortmund), Emil-Figge-Straße 42, Dortmund, 44227, Germany; Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Hufelandstraße 55, Essen, 45147, Germany
| | - Torsten Zesch
- Computational Linguistics, CATALPA - Center for Advanced Technology-Assisted Learning and Predictive Analytics, FernUniversität in Hagen, Germany
| |
Collapse
|
41
|
Mishra AK, Chappell MJ, Emerson S, Skubic M. Fall Risk Prediction in Older Adults Using Free-Text Nursing Notes and Medications in Electronic Health Records. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082830 DOI: 10.1109/embc40787.2023.10341127] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Nursing notes in Electronic Health Records (EHR) contain critical health information, including fall risk factors. However, an exploration of fall risk prediction using nursing notes is not well examined. In this study, we explored deep learning architectures to predict fall risk in older adults using text in nursing notes and medications in the EHR. EHR predictor data and fall events outcome data were obtained from 162 older adults living at TigerPlace, a senior living facility located in Columbia, MO. We used pre-trained BioWordVec embeddings to represent the words in the clinical notes and medications and trained multiple recurrent neural network-based natural language processing models to predict future fall events. Our final model predicted falls with an accuracy of 0.81, a sensitivity of 0.75, a specificity of 0.83, and an F1 score of 0.82. This preliminary exploratory analysis provides supporting evidence that fall risk can be predicted from clinical notes and medications. Future studies will utilize additional data modalities available in the EHR to potentially improve fall risk prediction from EHR data.
Collapse
|
42
|
Dolatabadi E, Chen B, Buchan SA, Austin AM, Azimaee M, McGeer A, Mubareka S, Kwong JC. Natural Language Processing for Clinical Laboratory Data Repository Systems: Implementation and Evaluation for Respiratory Viruses. JMIR AI 2023; 2:e44835. [PMID: 38875570 PMCID: PMC11057455 DOI: 10.2196/44835] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 03/31/2023] [Accepted: 04/18/2023] [Indexed: 06/16/2024]
Abstract
BACKGROUND With the growing volume and complexity of laboratory repositories, it has become tedious to parse unstructured data into structured and tabulated formats for secondary uses such as decision support, quality assurance, and outcome analysis. However, advances in natural language processing (NLP) approaches have enabled efficient and automated extraction of clinically meaningful medical concepts from unstructured reports. OBJECTIVE In this study, we aimed to determine the feasibility of using the NLP model for information extraction as an alternative approach to a time-consuming and operationally resource-intensive handcrafted rule-based tool. Therefore, we sought to develop and evaluate a deep learning-based NLP model to derive knowledge and extract information from text-based laboratory reports sourced from a provincial laboratory repository system. METHODS The NLP model, a hierarchical multilabel classifier, was trained on a corpus of laboratory reports covering testing for 14 different respiratory viruses and viral subtypes. The corpus includes 87,500 unique laboratory reports annotated by 8 subject matter experts (SMEs). The classification task involved assigning the laboratory reports to labels at 2 levels: 24 fine-grained labels in level 1 and 6 coarse-grained labels in level 2. A "label" also refers to the status of a specific virus or strain being tested or detected (eg, influenza A is detected). The model's performance stability and variation were analyzed across all labels in the classification task. Additionally, the model's generalizability was evaluated internally and externally on various test sets. RESULTS Overall, the NLP model performed well on internal, out-of-time (pre-COVID-19), and external (different laboratories) test sets with microaveraged F1-scores >94% across all classes. Higher precision and recall scores with less variability were observed for the internal and pre-COVID-19 test sets. As expected, the model's performance varied across categories and virus types due to the imbalanced nature of the corpus and sample sizes per class. There were intrinsically fewer classes of viruses being detected than those tested; therefore, the model's performance (lowest F1-score of 57%) was noticeably lower in the detected cases. CONCLUSIONS We demonstrated that deep learning-based NLP models are promising solutions for information extraction from text-based laboratory reports. These approaches enable scalable, timely, and practical access to high-quality and encoded laboratory data if integrated into laboratory information system repositories.
Collapse
Affiliation(s)
- Elham Dolatabadi
- Vector Institute, Toronto, ON, Canada
- School of Health Policy and Management, Faculty of Health, York University, Toronto, ON, Canada
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| | | | - Sarah A Buchan
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
- ICES, Toronto, ON, Canada
- Public Health Ontario, Toronto, ON, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
| | | | - Mahmoud Azimaee
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
- ICES, Toronto, ON, Canada
| | - Allison McGeer
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
- Sinai Health System, Toronto, ON, Canada
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada
| | - Samira Mubareka
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada
- Sunnybrook Research Institute, Toronto, ON, Canada
| | - Jeffrey C Kwong
- ICES, Toronto, ON, Canada
- Public Health Ontario, Toronto, ON, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
- University Health Network, Toronto, ON, Canada
- Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
43
|
Bikdeli B, Lo YC, Khairani CD, Bejjani A, Jimenez D, Barco S, Mahajan S, Caraballo C, Secemsky EA, Klok FA, Hunsaker AR, Aghayev A, Muriel A, Wang Y, Hussain MA, Appah-Sampong A, Lu Y, Lin Z, Aneja S, Khera R, Goldhaber SZ, Zhou L, Monreal M, Krumholz HM, Piazza G. Developing Validated Tools to Identify Pulmonary Embolism in Electronic Databases: Rationale and Design of the PE-EHR+ Study. Thromb Haemost 2023; 123:649-662. [PMID: 36809777 PMCID: PMC11200175 DOI: 10.1055/a-2039-3222] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/23/2023]
Abstract
BACKGROUND Contemporary pulmonary embolism (PE) research, in many cases, relies on data from electronic health records (EHRs) and administrative databases that use International Classification of Diseases (ICD) codes. Natural language processing (NLP) tools can be used for automated chart review and patient identification. However, there remains uncertainty with the validity of ICD-10 codes or NLP algorithms for patient identification. METHODS The PE-EHR+ study has been designed to validate ICD-10 codes as Principal Discharge Diagnosis, or Secondary Discharge Diagnoses, as well as NLP tools set out in prior studies to identify patients with PE within EHRs. Manual chart review by two independent abstractors by predefined criteria will be the reference standard. Sensitivity, specificity, and positive and negative predictive values will be determined. We will assess the discriminatory function of code subgroups for intermediate- and high-risk PE. In addition, accuracy of NLP algorithms to identify PE from radiology reports will be assessed. RESULTS A total of 1,734 patients from the Mass General Brigham health system have been identified. These include 578 with ICD-10 Principal Discharge Diagnosis codes for PE, 578 with codes in the secondary position, and 578 without PE codes during the index hospitalization. Patients within each group were selected randomly from the entire pool of patients at the Mass General Brigham health system. A smaller subset of patients will also be identified from the Yale-New Haven Health System. Data validation and analyses will be forthcoming. CONCLUSIONS The PE-EHR+ study will help validate efficient tools for identification of patients with PE in EHRs, improving the reliability of efficient observational studies or randomized trials of patients with PE using electronic databases.
Collapse
Affiliation(s)
- Behnood Bikdeli
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Cardiovascular Research Foundation (CRF), New York, New York, United States
| | - Ying-Chih Lo
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Candrika D Khairani
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Antoine Bejjani
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - David Jimenez
- Respiratory Department, Hospital Ramón y Cajal and Medicine Department, Universidad de Alcalá (Instituto de Ramón y Cajal de Investigación Sanitaria), Centro de Investigación Biomédica en Red de Enfermedades Respiratorias, Madrid, Spain
| | - Stefano Barco
- Department of Angiology, University Hospital Zurich, Zurich, Switzerland
- Center for Thrombosis and Hemostasis, Johannes Gutenberg University Mainz, Mainz, Germany
| | - Shiwani Mahajan
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, United States
| | - César Caraballo
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
| | - Eric A Secemsky
- Richard A. and Susan F. Smith Center for Outcomes Research in Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
- Harvard Medical School, Boston, Massachusetts, United States
- Division of Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
| | - Frederikus A Klok
- Department of Medicine - Thrombosis and Hemostasis, Leiden University Medical Centre, Leiden, The Netherlands
| | - Andetta R Hunsaker
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Ayaz Aghayev
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Alfonso Muriel
- Clinical Biostatistics Unit. Hospital Universitario Ramón y Cajal. IRYCIS, CIBERESP: Universidad de Alcalá. Madrid, Spain
| | - Yun Wang
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Richard A. and Susan F. Smith Center for Outcomes Research in Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States
| | - Mohamad A Hussain
- Division of Vascular and Endovascular Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Centre for Surgery and Public Health, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Abena Appah-Sampong
- Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, United States
| | - Yuan Lu
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
| | - Zhenqiu Lin
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
| | - Sanjay Aneja
- Department of Therapeutic Radiology, Yale School of Medicine, New Haven, Connecticut, United States
| | - Rohan Khera
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, United States
| | - Samuel Z Goldhaber
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Li Zhou
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Manuel Monreal
- Cátedra de Enfermedad Tromboembólica, Universidad Católica de Murcia, Murcia, Spain
| | - Harlan M Krumholz
- YNHH/Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, United States
- Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut, United States
| | - Gregory Piazza
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
- Thrombosis Research Group, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| |
Collapse
|
44
|
Yang L, Huang X, Wang J, Yang X, Ding L, Li Z, Li J. Identifying stroke-related quantified evidence from electronic health records in real-world studies. Artif Intell Med 2023; 140:102552. [PMID: 37210153 DOI: 10.1016/j.artmed.2023.102552] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2021] [Revised: 02/28/2023] [Accepted: 04/11/2023] [Indexed: 05/22/2023]
Abstract
BACKGROUND Stroke is one of the leading causes of death and disability worldwide. The National Institutes of Health Stroke Scale (NIHSS) scores in electronic health records (EHRs), which quantitatively describe patients' neurological deficits in evidence-based treatment, are crucial in stroke-related clinical investigations. However, the free-text format and lack of standardization inhibit their effective use. Automatically extracting the scale scores from the clinical free text so that its potential value in real-world studies is realized has become an important goal. OBJECTIVE This study aims to develop an automated method to extract scale scores from the free text of EHRs. METHODS We propose a two-step pipeline method to identify NIHSS items and numerical scores and validate its feasibility using a freely accessible critical care database: MIMIC-III (Medical Information Mart for Intensive Care III). First, we utilize MIMIC-III to create an annotated corpus. Then, we investigate possible machine learning methods for two subtasks, NIHSS item and score recognition and item-score relation extraction. In the evaluation, we conduct both task-specific and end-to-end evaluations and compare our method with the rule-based method using precision, recall and F1 scores as evaluation metrics. RESULTS We use all available discharge summaries of stroke cases in MIMIC-III. The annotated NIHSS corpus contains 312 cases, 2929 scale items, 2774 scores and 2733 relations. The results show that the best F1-score of our method was 0.9006, which was attained by combining BERT-BiLSTM-CRF and Random Forest, and it outperformed the rule-based method (F1-score = 0.8098). In the end-to-end task, our method could successfully recognize the item "1b level of consciousness questions", the score "1" and their relation "('1b level of consciousness questions', '1', 'has value')" from the sentence "1b level of consciousness questions: said name = 1", while the rule-based method could not. CONCLUSIONS The two-step pipeline method we propose is an effective approach to identify NIHSS items, scores and their relations. With its help, clinical investigators can easily retrieve and access structured scale data, thereby supporting stroke-related real-world studies.
Collapse
Affiliation(s)
- Lin Yang
- Institute of Medical Information and Library, Chinese Academy of Medical Sciences/Peking Union Medical College, Beijing 100020, China; Key Laboratory of Medical Information Intelligent Technology, Chinese Academy of Medical Sciences, Beijing 100020, China
| | - Xiaoshuo Huang
- Institute of Medical Information and Library, Chinese Academy of Medical Sciences/Peking Union Medical College, Beijing 100020, China; School of Health Care Technology, Dalian Neusoft University of Information, Dalian 116023, China
| | - Jiayang Wang
- Institute of Medical Information and Library, Chinese Academy of Medical Sciences/Peking Union Medical College, Beijing 100020, China
| | - Xin Yang
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China; National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Lingling Ding
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China; Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Zixiao Li
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China; National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China; Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Jiao Li
- Institute of Medical Information and Library, Chinese Academy of Medical Sciences/Peking Union Medical College, Beijing 100020, China; Key Laboratory of Medical Information Intelligent Technology, Chinese Academy of Medical Sciences, Beijing 100020, China.
| |
Collapse
|
45
|
Garg M, Sohn S. CareD: Caregiver's Experience with Cognitive Decline in Reddit Posts. IEEE INTERNATIONAL CONFERENCE ON HEALTHCARE INFORMATICS. IEEE INTERNATIONAL CONFERENCE ON HEALTHCARE INFORMATICS 2023; 2023:581-587. [PMID: 38384500 PMCID: PMC10877621 DOI: 10.1109/ichi57859.2023.00104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/23/2024]
Abstract
With advancements in analysis of cognitive decline in electronic health records, the research community witnesses a recent surge in social media posting by caregivers and/or loved ones of people with cognitive decline. The major challenges in this area are availability of large and diverse datasets, ethics of data collection and sharing, diagnostic specificity and clinical acceptability. To this end, we construct a new dataset, Caregivers experiences with cognitive Decline (CareD), of 1005 posts with more than 194K words and 9541 sentences, highlighting discussions on people with dementia and Alzheimer's disease on Reddit. We discuss the changing trends of discussions on cognitive decline in social media and open challenges for natural language processing and social computing. We first identify the Reddit posts reflecting substantial information as candidate posts. We further formulate the annotation guidelines, handle perplexities to investigate the existence of experiences, self-reported articles and potential caregiver in candidate posts, resulting in the discovery of latent symptoms, firsthand information, and prospective source of longitudinal information about the patient, respectively.
Collapse
Affiliation(s)
- Muskan Garg
- Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Sunghwan Sohn
- Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
46
|
Shelest-Szumilas O, Wozniak M. The Fears and Hopes of Ukrainian Migrant Workers in Poland in the Pandemic Era. JOURNAL OF INTERNATIONAL MIGRATION AND INTEGRATION 2023:1-23. [PMID: 37360639 PMCID: PMC10209937 DOI: 10.1007/s12134-023-01051-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/02/2023] [Indexed: 06/28/2023]
Abstract
Due to the COVID-19 pandemic, many immigrants found themselves in extremely unstable situations. The recent contributions show that employment decline in the first several months of the lockdown was higher for migrant workers than for natives. At the same time, migrants were less likely to find new employment in the recovery months. Such circumstances may result in an increased level of anxiety about one's economic situation. On the other hand, an unfavorable environment may induce resources that could help to overcome it. The paper aims to reveal migrants' concerns together with ambitions connected with the economic activity during the pandemic. The study is based on 30 individual in-depth interviews with Ukrainian migrant workers from Poland. The research approach was based on Natural Language Processing techniques. We employed sentiment analysis algorithms, and on a basis of selected lexicons, we extracted fears and hopes that appear in migrants' narrations. We also identified major topics and associated them with specific sentiments. Pandemic induced several matters connected with e.g., the stability of employment, discrimination, relationships, family, and financial situation. These affairs are usually connected on the basis of a cause-and-effect relationship. In addition, while several topics were common for both male and female participants, some of them were specific for each group.
Collapse
Affiliation(s)
- Olena Shelest-Szumilas
- Department of Education and Personnel Development, Poznan University of Economics and Business, Poznan, Poland
| | | |
Collapse
|
47
|
Rani S, Jain A. Optimizing healthcare system by amalgamation of text processing and deep learning: a systematic review. MULTIMEDIA TOOLS AND APPLICATIONS 2023:1-25. [PMID: 37362695 PMCID: PMC10183315 DOI: 10.1007/s11042-023-15539-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/23/2021] [Revised: 05/18/2022] [Accepted: 04/19/2023] [Indexed: 06/28/2023]
Abstract
The explosion of clinical textual data has drawn the attention of researchers. Owing to the abundance of clinical data, it is becoming difficult for healthcare professionals to take real-time measures. The tools and methods are lacking when compared to the amount of clinical data generated every day. This review aims to survey the text processing pipeline with deep learning methods such as CNN, RNN, LSTM, and GRU in the healthcare domain and discuss various applications such as clinical concept detection and extraction, medically aware dialogue systems, sentiment analysis of drug reviews shared online, clinical trial matching, and pharmacovigilance. In addition, we highlighted the major challenges in deploying text processing with deep learning to clinical textual data and identified the scope of research in this domain. Furthermore, we have discussed various resources that can be used in the future to optimize the healthcare domain by amalgamating text processing and deep learning.
Collapse
Affiliation(s)
- Somiya Rani
- Department of Computer Science and Engineering, NSUT East Campus (erstwhile AIACTR), Affiliated to Guru Gobind Singh Indraprastha University, Delhi, India
| | - Amita Jain
- Department of Computer Science and Engineering, Netaji Subhas University of Technology, Delhi, India
| |
Collapse
|
48
|
Kablan R, Miller HA, Suliman S, Frieboes HB. Evaluation of stacked ensemble model performance to predict clinical outcomes: A COVID-19 study. Int J Med Inform 2023; 175:105090. [PMID: 37172507 PMCID: PMC10165871 DOI: 10.1016/j.ijmedinf.2023.105090] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2023] [Revised: 04/17/2023] [Accepted: 05/03/2023] [Indexed: 05/15/2023]
Abstract
BACKGROUND The application of machine learning (ML) to analyze clinical data with the goal to predict patient outcomes has garnered increasing attention. Ensemble learning has been used in conjunction with ML to improve predictive performance. Although stacked generalization (stacking), a type of heterogeneous ensemble of ML models, has emerged in clinical data analysis, it remains unclear how to define the best model combinations for strong predictive performance. This study develops a methodology to evaluate the performance of "base" learner models and their optimized combination using "meta" learner models in stacked ensembles to accurately assess performance in the context of clinical outcomes. METHODS De-identified COVID-19 data was obtained from the University of Louisville Hospital, where a retrospective chart review was performed from March 2020 to November 2021. Three differently-sized subsets using features from the overall dataset were chosen to train and evaluate ensemble classification performance. The number of base learners chosen from several algorithm families coupled with a complementary meta learner was varied from a minimum of 2 to a maximum of 8. Predictive performance of these combinations was evaluated in terms of mortality and severe cardiac event outcomes using area-under-the-receiver-operating-characteristic (AUROC), F1, balanced accuracy, and kappa. RESULTS The results highlight the potential to accurately predict clinical outcomes, such as severe cardiac events with COVID-19, from routinely acquired in-hospital patient data. Meta learners Generalized Linear Model (GLM), Multi-Layer Perceptron (MLP), and Partial Least Squares (PLS) had the highest AUROC for both outcomes, while K-Nearest Neighbors (KNN) had the lowest. Performance trended lower in the training set as the number of features increased, and exhibited less variance in both training and validation across all feature subsets as the number of base learners increased. CONCLUSION This study offers a methodology to robustly evaluate ensemble ML performance when analyzing clinical data.
Collapse
Affiliation(s)
- Rianne Kablan
- Department of Bioengineering, University of Louisville, Louisville, KY, USA
| | - Hunter A Miller
- Department of Bioengineering, University of Louisville, Louisville, KY, USA
| | | | - Hermann B Frieboes
- Department of Bioengineering, University of Louisville, Louisville, KY, USA; James Graham Brown Cancer Center, University of Louisville, Louisville, KY, USA; Center for Predictive Medicine, University of Louisville, Louisville, KY, USA.
| |
Collapse
|
49
|
Bean DM, Kraljevic Z, Shek A, Teo J, Dobson RJB. Hospital-wide natural language processing summarising the health data of 1 million patients. PLOS DIGITAL HEALTH 2023; 2:e0000218. [PMID: 37159441 PMCID: PMC10168555 DOI: 10.1371/journal.pdig.0000218] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/28/2022] [Accepted: 02/16/2023] [Indexed: 05/11/2023]
Abstract
Electronic health records (EHRs) represent a major repository of real world clinical trajectories, interventions and outcomes. While modern enterprise EHR's try to capture data in structured standardised formats, a significant bulk of the available information captured in the EHR is still recorded only in unstructured text format and can only be transformed into structured codes by manual processes. Recently, Natural Language Processing (NLP) algorithms have reached a level of performance suitable for large scale and accurate information extraction from clinical text. Here we describe the application of open-source named-entity-recognition and linkage (NER+L) methods (CogStack, MedCAT) to the entire text content of a large UK hospital trust (King's College Hospital, London). The resulting dataset contains 157M SNOMED concepts generated from 9.5M documents for 1.07M patients over a period of 9 years. We present a summary of prevalence and disease onset as well as a patient embedding that captures major comorbidity patterns at scale. NLP has the potential to transform the health data lifecycle, through large-scale automation of a traditionally manual task.
Collapse
Affiliation(s)
- Daniel M Bean
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
- Health Data Research UK London, University College London, London, United Kingdom
| | - Zeljko Kraljevic
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
- NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London, London, United Kingdom
| | - Anthony Shek
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
- Department of Clinical Neuroscience, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | - James Teo
- Department of Clinical Neuroscience, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
- Department of Neuroscience, King's College Hospital NHS Foundation Trust, London, United Kingdom
| | - Richard J B Dobson
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
- Health Data Research UK London, University College London, London, United Kingdom
- NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London, London, United Kingdom
- Institute for Health Informatics, University College London, London, United Kingdom
- NIHR Biomedical Research Centre, University College London Hospitals NHS Foundation Trust, London, United Kingdom
| |
Collapse
|
50
|
Büttner M, Schwendicke F. Natural language processing in dentistry. Br Dent J 2023; 234:753. [PMID: 37237206 DOI: 10.1038/s41415-023-5854-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Accepted: 04/03/2023] [Indexed: 05/28/2023]
Affiliation(s)
| | - Falk Schwendicke
- Professor and Head of Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Germany.
| |
Collapse
|