Deep Natural Language Processing to Identify Symptom Documentation in Clinical Notes for Patients With Heart Failure Undergoing Cardiac Resynchronization Therapy

Natural Language Processing Accurately Differentiates Cancer Symptom Information in Electronic Health Record Narratives

PURPOSE

Identifying cancer symptoms in electronic health record (EHR) narratives is feasible with natural language processing (NLP). However, more efficient NLP systems are needed to detect various symptoms and distinguish observed symptoms from negated symptoms and medication-related side effects. We evaluated the accuracy of NLP in (1) detecting 14 symptom groups (ie, pain, fatigue, swelling, depressed mood, anxiety, nausea/vomiting, pruritus, headache, shortness of breath, constipation, numbness/tingling, decreased appetite, impaired memory, disturbed sleep) and (2) distinguishing observed symptoms in EHR narratives among patients with cancer.

METHODS

We extracted 902,508 notes for 11,784 unique patients diagnosed with cancer and developed a gold standard corpus of 1,112 notes labeled for presence or absence of 14 symptom groups. We trained an embeddings-augmented NLP system integrating human and machine intelligence and conventional machine learning algorithms. NLP metrics were calculated on a gold standard corpus subset for testing.

RESULTS

The interannotator agreement for labeling the gold standard corpus was excellent at 92%. The embeddings-augmented NLP model achieved the best performance (F1 score = 0.877). The highest NLP accuracy was observed in pruritus (F1 score = 0.937) while the lowest accuracy was in swelling (F1 score = 0.787). After classifying the entire data set with embeddings-augmented NLP, we found that 41% of the notes included symptom documentation. Pain was the most documented symptom (29% of all notes) while impaired memory was the least documented (0.7% of all notes).

CONCLUSION

We illustrated the feasibility of detecting 14 symptom groups in EHR narratives and showed that an embeddings-augmented NLP system outperforms conventional machine learning algorithms in detecting symptom information and differentiating observed symptoms from negated symptoms and medication-related side effects.

Recent advancements and applications of deep learning in heart failure: Α systematic review

BACKGROUND

Heart failure (HF), a global health challenge, requires innovative diagnostic and management approaches. The rapid evolution of deep learning (DL) in healthcare necessitates a comprehensive review to evaluate these developments and their potential to enhance HF evaluation, aligning clinical practices with technological advancements.

OBJECTIVE

This review aims to systematically explore the contributions of DL technologies in the assessment of HF, focusing on their potential to improve diagnostic accuracy, personalize treatment strategies, and address the impact of comorbidities.

METHODS

A thorough literature search was conducted across four major electronic databases: PubMed, Scopus, Web of Science and IEEE Xplore, yielding 137 articles that were subsequently categorized into five primary application areas: cardiovascular disease (CVD) classification, HF detection, image analysis, risk assessment, and other clinical analyses. The selection criteria focused on studies utilizing DL algorithms for HF assessment, not limited to HF detection but extending to any attempt in analyzing and interpreting HF-related data.

RESULTS

The analysis revealed a notable emphasis on CVD classification and HF detection, with DL algorithms showing significant promise in distinguishing between affected individuals and healthy subjects. Furthermore, the review highlights DL's capacity to identify underlying cardiomyopathies and other comorbidities, underscoring its utility in refining diagnostic processes and tailoring treatment plans to individual patient needs.

CONCLUSIONS

This review establishes DL as a key innovation in HF management, highlighting its role in advancing diagnostic accuracy and personalized care. The insights provided advocate for the integration of DL in clinical settings and suggest directions for future research to enhance patient outcomes in HF care.

Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores

Importance

Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR.

Objective

Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates.

Methods

Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation.

Results

For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date.

Conclusions

In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Natural Language Processing in Electronic Health Records in relation to healthcare decision-making: A systematic review

BACKGROUND

Natural Language Processing (NLP) is widely used to extract clinical insights from Electronic Health Records (EHRs). However, the lack of annotated data, automated tools, and other challenges hinder the full utilisation of NLP for EHRs. Various Machine Learning (ML), Deep Learning (DL) and NLP techniques are studied and compared to understand the limitations and opportunities in this space comprehensively.

METHODOLOGY

After screening 261 articles from 11 databases, we included 127 papers for full-text review covering seven categories of articles: (1) medical note classification, (2) clinical entity recognition, (3) text summarisation, (4) deep learning (DL) and transfer learning architecture, (5) information extraction, (6) Medical language translation and (7) other NLP applications. This study follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

RESULT AND DISCUSSION

EHR was the most commonly used data type among the selected articles, and the datasets were primarily unstructured. Various ML and DL methods were used, with prediction or classification being the most common application of ML or DL. The most common use cases were: the International Classification of Diseases, Ninth Revision (ICD-9) classification, clinical note analysis, and named entity recognition (NER) for clinical descriptions and research on psychiatric disorders.

CONCLUSION

We find that the adopted ML models were not adequately assessed. In addition, the data imbalance problem is quite important, yet we must find techniques to address this underlining problem. Future studies should address key limitations in studies, primarily identifying Lupus Nephritis, Suicide Attempts, perinatal self-harmed and ICD-9 classification.

Machine learning models to detect social distress, spiritual pain, and severe physical psychological symptoms in terminally ill patients with cancer from unstructured text data in electronic medical records

BACKGROUND

Few studies have developed automatic systems for identifying social distress, spiritual pain, and severe physical and phycological symptoms from text data in electronic medical records.

AIM

To develop models to detect social distress, spiritual pain, and severe physical and psychological symptoms in terminally ill patients with cancer from unstructured text data contained in electronic medical records.

DESIGN

A retrospective study of 1,554,736 narrative clinical records was analyzed 1 month before patients died. Supervised machine learning models were trained to detect comprehensive symptoms, and the performance of the models was tested using the area under the receiver operating characteristic curve (AUROC) and precision recall curve (AUPRC).

SETTING/PARTICIPANTS

A total of 808 patients was included in the study using records obtained from a university hospital in Japan between January 1, 2018 and December 31, 2019. As training data, we used medical records labeled for detecting social distress (n = 10,000) and spiritual pain (n = 10,000), and records that could be combined with the Support Team Assessment Schedule (based on date) for detecting severe physical/psychological symptoms (n = 5409).

RESULTS

Machine learning models for detecting social distress had AUROC and AUPRC values of 0.98 and 0.61, respectively; values for spiritual pain, were 0.90 and 0.58, respectively. The machine learning models accurately identified severe symptoms (pain, dyspnea, nausea, insomnia, and anxiety) with a high level of discrimination (AUROC > 0.8).

CONCLUSION

The machine learning models could detect social distress, spiritual pain, and severe symptoms in terminally ill patients with cancer from text data contained in electronic medical records.

Multi-label classification of symptom terms from free-text bilingual adverse drug reaction reports using natural language processing

Allergic reactions to medication range from mild to severe or even life-threatening. Proper documentation of patient allergy information is critical for safe prescription, avoiding drug interactions, and reducing healthcare costs. Allergy information is regularly obtained during the medical interview, but is often poorly documented in electronic health records (EHRs). While many EHRs allow for structured adverse drug reaction (ADR) reporting, a free-text entry is still common. The resulting information is neither interoperable nor easily reusable for other applications, such as clinical decision support systems and prescription alerts. Current approaches require pharmacists to review and code ADRs documented by healthcare professionals. Recently, the effectiveness of machine algorithms in natural language processing (NLP) has been widely demonstrated. Our study aims to develop and evaluate different NLP algorithms that can encode unstructured ADRs stored in EHRs into institutional symptom terms. Our dataset consists of 79,712 pharmacist-reviewed drug allergy records. We evaluated three NLP techniques: Naive Bayes—Support Vector Machine (NB-SVM), Universal Language Model Fine-tuning (ULMFiT), and Bidirectional Encoder Representations from Transformers (BERT). We tested different general-domain pre-trained BERT models, including mBERT, XLM-RoBERTa, and WanchanBERTa, as well as our domain-specific AllergyRoBERTa, which was pre-trained from scratch on our corpus. Overall, BERT models had the highest performance. NB-SVM outperformed ULMFiT and BERT for several symptom terms that are not frequently coded. The ensemble model achieved an exact match ratio of 95.33%, a F₁ score of 98.88%, and a mean average precision of 97.07% for the 36 most frequently coded symptom terms. The model was then further developed into a symptom term suggestion system and achieved a Krippendorff’s alpha agreement coefficient of 0.7081 in prospective testing with pharmacists. Some degree of automation could both accelerate the availability of allergy information and reduce the efforts for human coding.

Automatic symptoms identification from a massive volume of unstructured medical consultations using deep neural and BERT models

Automatic symptom identification plays a crucial role in assisting doctors during the diagnosis process in Telemedicine. In general, physicians spend considerable time on clinical documentation and symptom identification, which is unfeasible due to their full schedule. With text-based consultation services in telemedicine, the identification of symptoms from a user's consultation is a sophisticated process and time-consuming. Moreover, at Altibbi, which is an Arabic telemedicine platform and the context of this work, users consult doctors and describe their conditions in different Arabic dialects which makes the problem more complex and challenging. Therefore, in this work, an advanced deep learning approach is developed consultations with multi-dialects. The approach is formulated as a multi-label multi-class classification using features extracted based on AraBERT and fine-tuned on the bidirectional long short-term memory (BiLSTM) network. The Fine-tuning of BiLSTM relies on features engineered based on different variants of the bidirectional encoder representations from transformers (BERT). Evaluating the models based on precision, recall, and a customized hit rate showed a successful identification of symptoms from Arabic texts with promising accuracy. Hence, this paves the way toward deploying an automated symptom identification model in production at Altibbi which can help general practitioners in telemedicine in providing more efficient and accurate consultations.

Identification of Uncontrolled Symptoms in Cancer Patients Using Natural Language Processing

CONTEXT

For patients with cancer, uncontrolled pain and other symptoms are the leading cause of unplanned hospitalizations. Early access to specialty palliative care (PC) is effective to reduce symptom burden, but more efficient approaches are needed for rapid identification and referral. Information on symptom burden largely exists in free-text notes, limiting its utility as a trigger for best practice alerts or automated referrals.

OBJECTIVES

To evaluate whether natural language processing (NLP) can be used to identify uncontrolled symptoms (pain, dyspnea, or nausea/vomiting) in the electronic health record (EHR) among hospitalized cancer patients with advanced disease.

METHODS

The dataset included 1,644 hospitalization encounters for cancer patients admitted from 1/2017 -6/2019. We randomly sampled 296 encounters, which included 15,580 clinical notes. We manually reviewed the notes and recorded symptom severity. The primary endpoint was an indicator for whether a symptom was labeled as "controlled" (none, mild, not reported) or as "uncontrolled" (moderate or severe). We randomly split the data into training and test sets and used the Random Forest algorithm to evaluate final model performance.

RESULTS

Our models predicted presence of an uncontrolled symptom with the following performance: pain with 61% accuracy, 69% sensitivity, and 46% specificity (F1: 69.5); nausea/vomiting with 68% accuracy, 21% sensitivity, and 90% specificity (F1: 29.4); and dyspnea with 80% accuracy, 22% sensitivity, and 88% specificity (F1: 21.1).

CONCLUSION

This study demonstrated initial feasibility of using NLP to identify hospitalized cancer patients with uncontrolled symptoms. Further model development is needed before these algorithms could be implemented to trigger early access to PC.

Natural language processing for automated surveillance of intraoperative neuromonitoring in spine surgery

We sought to develop natural language processing (NLP) methods for automated detection and characterization of neuromonitoring documentation from free-text operative reports in patients undergoing spine surgery. We included 13,718 patients who received spine surgery at two tertiary academic medical centers between December 2000 - December 2020. We first validated a rule-based NLP method for identifying operative reports containing neuromonitoring documentation, comparing performance to standard administrative codes. We then trained a deep learning model in a subset of 993 patients to characterize neuromonitoring documentation and identify events indicating change in status or difficulty establishing baseline signals. Performance of the deep learning model was compared to gold-standard manual chart review. In our patient population, 3,606 (26.3%) patients had neuromonitoring documentation identified using NLP. Our NLP method identified notes containing neuromonitoring documentation with an F1-score of 1.0, surpassing performance of standard administrative codes which had an F1-score of 0.64. In the subset of 993 patients used for training, validation, and testing a deep learning model, the prevalence of change in status was 6.5% and difficulty establishing neuromonitoring baseline signals was 6.6%. The deep learning model had an F1-score = 0.80 and AUC-ROC = 1.0 for identifying change in status, and an F1-score = 0.80 and AUC-ROC = 0.97 for identifying difficulty establishing baseline signals. Compared to gold standard manual chart review, our methodology has greater efficiency for identifying infrequent yet important types of neuromonitoring documentation. This method may facilitate large-scale quality improvement initiatives that require timely analysis of a large volume of EHRs.

Artificial Intelligence-Based Echocardiographic Left Atrial Volume Measurement with Pulmonary Vein Comparison

This paper combines echocardiographic signal processing and artificial intelligence technology to propose a deep neural network model adapted to echocardiographic signals to achieve left atrial volume measurement and automatic assessment of pulmonary veins efficiently and quickly. Based on the echocardiographic signal generation mechanism and detection method, an experimental scheme for the echocardiographic signal acquisition was designed. The echocardiographic signal data of healthy subjects were measured in four different experimental states, and a database of left atrial volume measurements and pulmonary veins was constructed. Combining the correspondence between ECG signals and echocardiographic signals in the time domain, a series of preprocessing such as denoising, feature point localization, and segmentation of the cardiac cycle was realized by wavelet transform and threshold method to complete the data collection. This paper proposes a comparative model based on artificial intelligence, adapts to the characteristics of one-dimensional time-series echocardiographic signals, automatically extracts the deep features of echocardiographic signals, effectively reduces the subjective influence of manual feature selection, and realizes the automatic classification and evaluation of human left atrial volume measurement and pulmonary veins under different states. The experimental results show that the proposed BP neural network model has good adaptability and classification performance in the tasks of LV volume measurement and pulmonary vein automatic classification evaluation and achieves an average test accuracy of over 96.58%. The average root-mean-square error percentage of signal compression is only 0.65% by extracting the coding features of the original echocardiographic signal through the convolutional autoencoder, which completes the signal compression with low loss. Comparing the training time and classification accuracy of the LSTM network with the original signal and encoded features, the experimental results show that the AI model can greatly reduce the model training time cost and achieve an average accuracy of 97.97% in the test set and increase the real-time performance of the left atrial volume measurement and pulmonary vein evaluation as well as the security of the data transmission process, which is very important for the comparison of left atrial volume measurement and pulmonary vein. It is of great practical importance to compare left atrial volume measurements with pulmonary veins.

Systematic review of current natural language processing methods and applications in cardiology

Natural language processing (NLP) is a set of automated methods to organise and evaluate the information contained in unstructured clinical notes, which are a rich source of real-world data from clinical care that may be used to improve outcomes and understanding of disease in cardiology. The purpose of this systematic review is to provide an understanding of NLP, review how it has been used to date within cardiology and illustrate the opportunities that this approach provides for both research and clinical care. We systematically searched six scholarly databases (ACM Digital Library, Arxiv, Embase, IEEE Explore, PubMed and Scopus) for studies published in 2015-2020 describing the development or application of NLP methods for clinical text focused on cardiac disease. Studies not published in English, lacking a description of NLP methods, non-cardiac focused and duplicates were excluded. Two independent reviewers extracted general study information, clinical details and NLP details and appraised quality using a checklist of quality indicators for NLP studies. We identified 37 studies developing and applying NLP in heart failure, imaging, coronary artery disease, electrophysiology, general cardiology and valvular heart disease. Most studies used NLP to identify patients with a specific diagnosis and extract disease severity using rule-based NLP methods. Some used NLP algorithms to predict clinical outcomes. A major limitation is the inability to aggregate findings across studies due to vastly different NLP methods, evaluation and reporting. This review reveals numerous opportunities for future NLP work in cardiology with more diverse patient samples, cardiac diseases, datasets, methods and applications.