1
|
Shyr C, Hu Y, Bastarache L, Cheng A, Hamid R, Harris P, Xu H. Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models. JOURNAL OF HEALTHCARE INFORMATICS RESEARCH 2024; 8:438-461. [PMID: 38681753 PMCID: PMC11052982 DOI: 10.1007/s41666-023-00155-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 10/24/2023] [Accepted: 11/13/2023] [Indexed: 05/01/2024]
Abstract
Purpose Phenotyping is critical for informing rare disease diagnosis and treatment, but disease phenotypes are often embedded in unstructured text. While natural language processing (NLP) can automate extraction, a major bottleneck is developing annotated corpora. Recently, prompt learning with large language models (LLMs) has been shown to lead to generalizable results without any (zero-shot) or few annotated samples (few-shot), but none have explored this for rare diseases. Our work is the first to study prompt learning for identifying and extracting rare disease phenotypes in the zero- and few-shot settings. Methods We compared the performance of prompt learning with ChatGPT and fine-tuning with BioClinicalBERT. We engineered novel prompts for ChatGPT to identify and extract rare diseases and their phenotypes (e.g., diseases, symptoms, and signs), established a benchmark for evaluating its performance, and conducted an in-depth error analysis. Results Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.610 in the zero- and few-shot settings, respectively). However, ChatGPT achieved higher accuracy for rare diseases and signs in the one-shot setting (F1 of 0.778 and 0.725). Conversational, sentence-based prompts generally achieved higher accuracy than structured lists. Conclusion Prompt learning using ChatGPT has the potential to match or outperform fine-tuning BioClinicalBERT at extracting rare diseases and signs with just one annotated sample. Given its accessibility, ChatGPT could be leveraged to extract these entities without relying on a large, annotated corpus. While LLMs can support rare disease phenotyping, researchers should critically evaluate model outputs to ensure phenotyping accuracy.
Collapse
Affiliation(s)
- Cathy Shyr
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203 USA
| | - Yan Hu
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77225 USA
| | - Lisa Bastarache
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203 USA
| | - Alex Cheng
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203 USA
| | - Rizwan Hamid
- Division of Medical Genetics and Genomic Medicine, Vanderbilt University Medical Center, Nashville, TN 37203 USA
| | - Paul Harris
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203 USA
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203 USA
- Department of Biomedical Engineering, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, TN 37203 USA
| | - Hua Xu
- Section of Biomedical Informatics and Data Science, Yale School of Medicine, 100 College Street, New Haven, CT 06510 USA
| |
Collapse
|
2
|
Hens D, Wyers L, Claeys KG. Validation of an Artificial Intelligence driven framework to automatically detect red flag symptoms in screening for rare diseases in electronic health records: hereditary transthyretin amyloidosis polyneuropathy as a key example. J Peripher Nerv Syst 2023; 28:79-85. [PMID: 36468607 DOI: 10.1111/jns.12523] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Revised: 11/19/2022] [Accepted: 11/28/2022] [Indexed: 12/07/2022]
Abstract
Rare life-threatening conditions, such as multisystemic hereditary transthyretin amyloidosis (ATTRv) polyneuropathy, are often underdiagnosed or diagnosed late in the disease course, although early diagnosis is crucial for treatment success. Red flag symptoms have been identified, but manual screening of multidisciplinary medical records on this set of symptoms is time-consuming. This study aimed to validate a Natural Language Processing (NLP) algorithm to perform such a search in an automated manner, in order to improve early diagnosis and treatment. A novel state-of-the-art NLP procedure was applied to extract red flag symptoms from patients' electronic medical records and to select patients at risk for ATTRv polyneuropathy for further clinical review. Accuracy of the algorithm was assessed through comparison with a manual standard on a random sample of 300 patients. Out of a retrospective sample of 1015 patients, the NLP algorithm yielded 128 patients with three or more red flag symptoms of which 69 patients were considered eligible for genetic testing after clinical review. High accuracy was found in the detection of red flag symptoms, with F1 scores between 0.88 and 0.98. A relative increase of 48.6% in genetic testing, to identify patients with a rare disease earlier, was demonstrated. An NLP algorithm, after clinical validation, offers a valid and accurate tool to detect red flag symptoms in medical records across multiple disciplines, supporting better screening for patients with rare diseases. This opens the door to further NLP applications, facilitating rapid diagnosis and early treatment of rare diseases.
Collapse
Affiliation(s)
| | | | - Kristl G Claeys
- Department of Neurology, University Hospitals Leuven, Leuven, Belgium.,Laboratory for Muscle Diseases and Neuropathies, Department of Neurosciences, KU Leuven, and Leuven Brain Institute (LBI), Leuven, Belgium
| |
Collapse
|
3
|
Negation-based transfer learning for improving biomedical Named Entity Recognition and Relation Extraction. J Biomed Inform 2023; 138:104279. [PMID: 36610608 DOI: 10.1016/j.jbi.2022.104279] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2022] [Revised: 12/03/2022] [Accepted: 12/30/2022] [Indexed: 01/06/2023]
Abstract
BACKGROUND AND OBJECTIVES Named Entity Recognition (NER) and Relation Extraction (RE) are two of the most studied tasks in biomedical Natural Language Processing (NLP). The detection of specific terms and entities and the relationships between them are key aspects for the development of more complex automatic systems in the biomedical field. In this work, we explore transfer learning techniques for incorporating information about negation into systems performing NER and RE. The main purpose of this research is to analyse to what extent the successful detection of negated entities in separate tasks helps in the detection of biomedical entities and their relationships. METHODS Three neural architectures are proposed in this work, all of them mainly based on Bidirectional Long Short-Term Memory (Bi-LSTM) networks and Conditional Random Fields (CRFs). While the first architecture is devoted to detecting triggers and scopes of negated entities in any domain, two specific models are developed for performing isolated NER tasks and joint NER and RE tasks in the biomedical domain. Then, weights related to negation detection learned by the first architecture are incorporated into those last models. Two different languages, Spanish and English, are taken into account in the experiments. RESULTS Performance of the biomedical models is analysed both when the weights of the neural networks are randomly initialized, and when weights from the negation detection model are incorporated into them. Improvements of around 3.5% of F-Measure in the English language and more than 7% in the Spanish language are achieved in the NER task, while the NER+RE task increases F-Measure scores by more than 13% for the NER submodel and around 2% for the RE submodel. CONCLUSIONS The obtained results allow us to conclude that negation-based transfer learning techniques are appropriate for performing biomedical NER and RE tasks. These results highlight the importance of detecting negation for improving the identification of biomedical entities and their relationships. The explored techniques show robustness by maintaining consistent results and improvements across different tasks and languages.
Collapse
|
4
|
Lee J, Liu C, Kim J, Chen Z, Sun Y, Rogers JR, Chung WK, Weng C. Deep learning for rare disease: A scoping review. J Biomed Inform 2022; 135:104227. [DOI: 10.1016/j.jbi.2022.104227] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 08/22/2022] [Accepted: 10/07/2022] [Indexed: 10/31/2022]
|
5
|
Segura-Bedmar I, Camino-Perdones D, Guerrero-Aspizua S. Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts. BMC Bioinformatics 2022; 23:263. [PMID: 35794528 PMCID: PMC9258216 DOI: 10.1186/s12859-022-04810-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2022] [Accepted: 06/21/2022] [Indexed: 11/10/2022] Open
Abstract
Abstract
Background and objective
Although rare diseases are characterized by low prevalence, approximately 400 million people are affected by a rare disease. The early and accurate diagnosis of these conditions is a major challenge for general practitioners, who do not have enough knowledge to identify them. In addition to this, rare diseases usually show a wide variety of manifestations, which might make the diagnosis even more difficult. A delayed diagnosis can negatively affect the patient’s life. Therefore, there is an urgent need to increase the scientific and medical knowledge about rare diseases. Natural Language Processing (NLP) and Deep Learning can help to extract relevant information about rare diseases to facilitate their diagnosis and treatments.
Methods
The paper explores several deep learning techniques such as Bidirectional Long Short Term Memory (BiLSTM) networks or deep contextualized word representations based on Bidirectional Encoder Representations from Transformers (BERT) to recognize rare diseases and their clinical manifestations (signs and symptoms).
Results
BioBERT, a domain-specific language representation based on BERT and trained on biomedical corpora, obtains the best results with an F1 of 85.2% for rare diseases. Since many signs are usually described by complex noun phrases that involve the use of use of overlapped, nested and discontinuous entities, the model provides lower results with an F1 of 57.2%.
Conclusions
While our results are promising, there is still much room for improvement, especially with respect to the identification of clinical manifestations (signs and symptoms).
Collapse
|
6
|
Martínez-deMiguel C, Segura-Bedmar I, Chacón-Solano E, Guerrero-Aspizua S. The RareDis corpus: A corpus annotated with rare diseases, their signs and symptoms. J Biomed Inform 2021; 125:103961. [PMID: 34879250 DOI: 10.1016/j.jbi.2021.103961] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2021] [Revised: 11/08/2021] [Accepted: 11/22/2021] [Indexed: 11/26/2022]
Abstract
Rare diseases affect a small number of people compared to the general population. However, more than 6,000 different rare diseases exist and, in total, they affect more than 300 million people worldwide. Rare diseases share as part of their main problem, the delay in diagnosis and the sparse information available for researchers, clinicians, and patients. Finding a diagnostic can be a very long and frustrating experience for patients and their families. The average diagnostic delay is between 6-8 years. Many of these diseases result in different manifestations among patients, which hampers even more their detection and the correct treatment choice. Therefore, there is an urgent need to increase the scientific and medical knowledge about rare diseases. Natural Language Processing (NLP) can help to extract relevant information about rare diseases to facilitate their diagnosis and treatments, but most NLP techniques require manually annotated corpora. Therefore, our goal is to create a gold standard corpus annotated with rare diseases and their clinical manifestations. It could be used to train and test NLP approaches and the information extracted through NLP could enrich the knowledge of rare diseases, and thereby, help to reduce the diagnostic delay and improve the treatment of rare diseases. The paper describes the selection of 1,041 texts to be included in the corpus, the annotation process and the annotation guidelines. The entities (disease, rare disease, symptom, sign and anaphor) and the relationships (produces, is a, is acron, is synon, increases risk of, anaphora) were annotated. The RareDis corpus contains more than 5,000 rare diseases and almost 6,000 clinical manifestations are annotated. Moreover, the Inter Annotator Agreement evaluation shows a relatively high agreement (F1-measure equal to 83.5% under exact match criteria for the entities and equal to 81.3% for the relations). Based on these results, this corpus is of high quality, supposing a significant step for the field since there is a scarcity of available corpus annotated with rare diseases. This could open the door to further NLP applications, which would facilitate the diagnosis and treatment of these rare diseases and, therefore, would improve dramatically the quality of life of these patients.
Collapse
Affiliation(s)
- Claudia Martínez-deMiguel
- Tissue Engineering and Regenerative Medicine group, Department of Bioengineering, Universidad Carlos III de Madrid, Avenidad de la Universidad, 30, Leganés 28911, Madrid, Spain
| | - Isabel Segura-Bedmar
- Human Language and Accesibility Technologies, Computer Science Department, Avenidad de la Universidad 30, Leganés 28911, Madrid, Spain.
| | - Esteban Chacón-Solano
- Tissue Engineering and Regenerative Medicine group, Department of Bioengineering, Universidad Carlos III de Madrid, Avenidad de la Universidad, 30, Leganés 28911, Madrid, Spain; Hospital Fundación Jiménez Díaz e, Instituto de Investigación, FJD, Av. de los Reyes Católicos 2, Madrid 28040, Madrid, Spain; Epithelial Biomedicine Division, CIEMAT, Madrid 28040, Madrid, Spain
| | - Sara Guerrero-Aspizua
- Tissue Engineering and Regenerative Medicine group, Department of Bioengineering, Universidad Carlos III de Madrid, Avenidad de la Universidad, 30, Leganés 28911, Madrid, Spain; Hospital Fundación Jiménez Díaz e, Instituto de Investigación, FJD, Av. de los Reyes Católicos 2, Madrid 28040, Madrid, Spain; Epithelial Biomedicine Division, CIEMAT, Madrid 28040, Madrid, Spain; Centre for Biomedical Network Research on Rare Diseases (CIBERER), C/Monforte de Lemos 3-5, Madrid 28029, Madrid, Spain
| |
Collapse
|
7
|
El-Allaly ED, Sarrouti M, En-Nahnahi N, Ouatik El Alaoui S. An adverse drug effect mentions extraction method based on weighted online recurrent extreme learning machine. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2019; 176:33-41. [PMID: 31200909 DOI: 10.1016/j.cmpb.2019.04.029] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/06/2019] [Revised: 04/05/2019] [Accepted: 04/26/2019] [Indexed: 06/09/2023]
Abstract
BACKGROUND AND OBJECTIVE Automatic extraction of adverse drug effect (ADE) mentions from biomedical texts is a challenging research problem that has attracted significant attention from the pharmacovigilance and biomedical text mining communities. Indeed, deep learning based methods have recently been employed to solve this issue with great success. However, they fail to effectively identify the boundary of mentions. In this paper, we propose a weighted online recurrent extreme learning machine (WOR-ELM) based method to overcome this drawback. METHODS The proposed method for ADE mentions extraction from biomedical texts is divided into two stages: span detection and ADE mentions classification. At the first stage, we identify the boundary of the mentions irrespective of their types with a WOR-ELM in a given sentence. At the second stage, another WOR-ELM is used to classify the identified mentions to the appropriate type. Both stages use the concatenation of character-level and word-level embeddings as features. The character-level embedding is obtained using a modified online recurrent extreme learning machine, whereas the word-level embedding is obtained from a pre-trained model. RESULTS Several experiments were carried out on a well-known ADE corpus to evaluate the effectiveness and demonstrate the usefulness of the proposed method. The obtained results show that our method achieves an F-score of 87.5%, which outperforms the current state-of-the-art methods. CONCLUSIONS Our research results indicate that the proposed method for adverse drug effect mentions extraction from text can significantly improve performance over existing methods. Our experiments show the effectiveness of incorporating word-level and character level embeddings as features for WOR-ELM. They also illustrate the benefits of using IOU segment to represent ADE mentions.
Collapse
Affiliation(s)
- Ed-Drissiya El-Allaly
- Laboratory of Informatics and Modeling, FSDM, Sidi Mohammed Ben Abdellah University, Fez, Morocco.
| | - Mourad Sarrouti
- Laboratory of Informatics and Modeling, FSDM, Sidi Mohammed Ben Abdellah University, Fez, Morocco.
| | - Noureddine En-Nahnahi
- Laboratory of Informatics and Modeling, FSDM, Sidi Mohammed Ben Abdellah University, Fez, Morocco.
| | - Said Ouatik El Alaoui
- Laboratory of Informatics and Modeling, FSDM, Sidi Mohammed Ben Abdellah University, Fez, Morocco.
| |
Collapse
|
8
|
Bonanni L. The democratic aspect of machine learning: Limitations and opportunities for Parkinson's disease. Mov Disord 2018; 34:164-166. [DOI: 10.1002/mds.27600] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2018] [Revised: 12/03/2018] [Accepted: 12/06/2018] [Indexed: 12/13/2022] Open
Affiliation(s)
- Laura Bonanni
- Department of Neuroscience, Imaging and Clinical SciencesUniversity G. d'Annunzio of Chieti‐Pescara Chieti Italy
| |
Collapse
|