Development and validation of a pragmatic natural language processing approach to identifying falls in older adults in the emergency department

The use of natural language processing for the identification of ageing syndromes including sarcopenia, frailty and falls in electronic healthcare records: a systematic review

BACKGROUND

Recording and coding of ageing syndromes in hospital records is known to be suboptimal. Natural Language Processing algorithms may be useful to identify diagnoses in electronic healthcare records to improve the recording and coding of these ageing syndromes, but the feasibility and diagnostic accuracy of such algorithms are unclear.

METHODS

We conducted a systematic review according to a predefined protocol and in line with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Searches were run from the inception of each database to the end of September 2023 in PubMed, Medline, Embase, CINAHL, ACM digital library, IEEE Xplore and Scopus. Eligible studies were identified via independent review of search results by two coauthors and data extracted from each study to identify the computational method, source of text, testing strategy and performance metrics. Data were synthesised narratively by ageing syndrome and computational method in line with the Studies Without Meta-analysis guidelines.

RESULTS

From 1030 titles screened, 22 studies were eligible for inclusion. One study focussed on identifying sarcopenia, one frailty, twelve falls, five delirium, five dementia and four incontinence. Sensitivity (57.1%-100%) of algorithms compared with a reference standard was reported in 20 studies, and specificity (84.0%-100%) was reported in only 12 studies. Study design quality was variable with results relevant to diagnostic accuracy not always reported, and few studies undertaking external validation of algorithms.

CONCLUSIONS

Current evidence suggests that Natural Language Processing algorithms can identify ageing syndromes in electronic health records. However, algorithms require testing in rigorously designed diagnostic accuracy studies with appropriate metrics reported.

Using natural language processing in emergency medicine health service research: A systematic review and meta-analysis

OBJECTIVES

Natural language processing (NLP) represents one of the adjunct technologies within artificial intelligence and machine learning, creating structure out of unstructured data. This study aims to assess the performance of employing NLP to identify and categorize unstructured data within the emergency medicine (EM) setting.

METHODS

We systematically searched publications related to EM research and NLP across databases including MEDLINE, Embase, Scopus, CENTRAL, and ProQuest Dissertations & Theses Global. Independent reviewers screened, reviewed, and evaluated article quality and bias. NLP usage was categorized into syndromic surveillance, radiologic interpretation, and identification of specific diseases/events/syndromes, with respective sensitivity analysis reported. Performance metrics for NLP usage were calculated and the overall area under the summary of receiver operating characteristic curve (SROC) was determined.

RESULTS

A total of 27 studies underwent meta-analysis. Findings indicated an overall mean sensitivity (recall) of 82%-87%, specificity of 95%, with the area under the SROC at 0.96 (95% CI 0.94-0.98). Optimal performance using NLP was observed in radiologic interpretation, demonstrating an overall mean sensitivity of 93% and specificity of 96%.

CONCLUSIONS

Our analysis revealed a generally favorable performance accuracy in using NLP within EM research, particularly in the realm of radiologic interpretation. Consequently, we advocate for the adoption of NLP-based research to augment EM health care management.

Improving postsurgical fall detection for older Americans using LLM-driven analysis of clinical narratives

Postsurgical falls have significant patient and societal implications but remain challenging to identify and track. Detecting postsurgical falls is crucial to improve patient care for older adults and reduce healthcare costs. Large language models (LLMs) offer a promising solution for reliable and automated fall detection using unstructured data in clinical notes. We tested several LLM prompting approaches to postsurgical fall detection in two different healthcare systems with three open-source LLMs. The Mixtral-8×7B zero-shot had the best performance at Stanford Health Care (PPV = 0.81, recall = 0.67) and the Veterans Health Administration (PPV = 0.93, recall = 0.94). These results demonstrate that LLMs can detect falls with little to no guidance and lay groundwork for applications of LLMs in fall prediction and prevention across many different settings.

Treatment characteristics among patients with binge-eating disorder: an electronic health records analysis

OBJECTIVES

Treatment for adults diagnosed with binge-eating disorder (BED) includes psychotherapy and/or pharmacotherapy and aims to reduce the frequency of binge-eating episodes and disordered eating, improve metabolic-related issues and reduce weight, and address mood symptoms. Data describing real-world treatment patterns are lacking; therefore, this study aims to characterize real-world treatment patterns among patients with BED.

METHODS

This retrospective study identified adult patients with BED using natural language processing of clinical notes from the Optum electronic health record database from 2009 to 2015. Treatment patterns were examined during the 12 months preceding the BED recognition date and during a follow-up period after BED recognition (1-3 years for most patients).

RESULTS

Among 1042 patients, 384 were categorized as the BED cohort and 658, who met less stringent criteria, were categorized as probable BED. In the BED cohort, mean ± SD age was 45.2 ± 13.4 years and 81.8% were women (probable BED, 45.9 ± 12.8 years, 80.2%). A greater percentage of patients in the BED cohort were prescribed pharmacotherapy (70.6% [probable BED, 66.9%]) than received/discussed psychotherapy (53.1% [probable BED, 39.2%]) at baseline. In the BED cohort, 54.4% of patients were prescribed antidepressants (probable BED, 52.4%), 25.3% stimulants (probable BED, 20.1%), and 34.4% nonspecific psychotherapy (probable BED, 24.6%) at baseline, with no substantive differences observed during follow-up. Low percentages of patients in the BED cohort received/discussed cognitive behavioral therapy at baseline (12.5% [probable BED, 9.0%) or during follow-up (13.0% [probable BED, 8.8%). Among patients with ≥1 psychotherapy visit, the mean ± SD number of visits in the BED cohort was 1.2 ± 5.9 at baseline (probable BED, 1.7 ± 7.3) and 2.2 ± 7.7 during follow-up (probable BED, 2.6 ± 7.7).

CONCLUSION

This cohort of patients with BED was treated more frequently with pharmacotherapy than psychotherapy. These data may help inform strategies for reducing differences between real-world treatment patterns and evidence-based recommendations.

Coding linguistic elements in clinical interactions: a step-by-step guide for analyzing communication form

Background

The quality of communication between healthcare professionals (HCPs) and patients affects health outcomes. Different coding systems have been developed to unravel the interaction. Most schemes consist of predefined categories that quantify the content of communication (the what). Though the form (the how) of the interaction is equally important, protocols that systematically code variations in form are lacking. Patterns of form and how they may differ between groups therefore remain unnoticed. To fill this gap, we present CLECI, Coding Linguistic Elements in Clinical Interactions, a protocol for the development of a quantitative codebook analyzing communication form in medical interactions.

Methods

Analyzing with a CLECI codebook is a four-step process, i.e. preparation, codebook development, (double-)coding, and analysis and report. Core activities within these phases are research question formulation, data collection, selection of utterances, iterative deductive and inductive category refinement, reliability testing, coding, analysis, and reporting.

Results and conclusion

We present step-by-step instructions for a CLECI analysis and illustrate this process in a case study. We highlight theoretical and practical issues as well as the iterative codebook development which combines theory-based and data-driven coding. Theory-based codes assess how relevant linguistic elements occur in natural interactions, whereas codes derived from the data accommodate linguistic elements to real-life interactions and contribute to theory-building. This combined approach increases research validity, enhances theory, and adjusts to fit naturally occurring data. CLECI will facilitate the study of communication form in clinical interactions and other institutional settings.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01647-0.

A hybrid model to identify fall occurrence from electronic health records

INTRODUCTION

Falls are a leading cause of unintentional injury in the elderly. Electronic health records (EHRs) offer the unique opportunity to develop models that can identify fall events. However, identifying fall events in clinical notes requires advanced natural language processing (NLP) to simultaneously address multiple issues because the word "fall" is a typical homonym.

METHODS

We implemented a context-aware language model, Bidirectional Encoder Representations from Transformers (BERT) to identify falls from the EHR text and further fused the BERT model into a hybrid architecture coupled with post-hoc heuristic rules to enhance the performance. The models were evaluated on real world EHR data and were compared to conventional rule-based and deep learning models (CNN and Bi-LSTM). To better understand the ability of each approach to identify falls, we further categorize fall-related concepts (i.e., risk of fall, prevention of fall, homonym) and performed a detailed error analysis.

RESULTS

The hybrid model achieved the highest f1-score on sentence (0.971), document (0.985), and patient (0.954) level. At the sentence level (basic data unit in the model), the hybrid model had 0.954, 1.000, 0.988, and 0.999 in sensitivity, specificity, positive predictive value, and negative predictive value, respectively. The error analysis showed that that machine learning-based approaches demonstrated higher performance than a rule-based approach in challenging cases that required contextual understanding. The context-aware language model (BERT) slightly outperformed the word embedding approach trained on Bi-LSTM. No single model yielded the best performance for all fall-related semantic categories.

CONCLUSION

A context-aware language model (BERT) was able to identify challenging fall events that requires context understanding in EHR free text. The hybrid model combined with post-hoc rules allowed a custom fix on the BERT outcomes and further improved the performance of fall detection.

Ambiguous and Incomplete: Natural Language Processing Reveals Problematic Reporting Styles in Thyroid Ultrasound Reports

OBJECTIVE

 Natural language processing (NLP) systems convert unstructured text into analyzable data. Here, we describe the performance measures of NLP to capture granular details on nodules from thyroid ultrasound (US) reports and reveal critical issues with reporting language.

METHODS

 We iteratively developed NLP tools using clinical Text Analysis and Knowledge Extraction System (cTAKES) and thyroid US reports from 2007 to 2013. We incorporated nine nodule features for NLP extraction. Next, we evaluated the precision, recall, and accuracy of our NLP tools using a separate set of US reports from an academic medical center (A) and a regional health care system (B) during the same period. Two physicians manually annotated each test-set report. A third physician then adjudicated discrepancies. The adjudicated "gold standard" was then used to evaluate NLP performance on the test-set.

RESULTS

 A total of 243 thyroid US reports contained 6,405 data elements. Inter-annotator agreement for all elements was 91.3%. Compared with the gold standard, overall recall of the NLP tool was 90%. NLP recall for thyroid lobe or isthmus characteristics was: laterality 96% and size 95%. NLP accuracy for nodule characteristics was: laterality 92%, size 92%, calcifications 76%, vascularity 65%, echogenicity 62%, contents 76%, and borders 40%. NLP recall for presence or absence of lymphadenopathy was 61%. Reporting style accounted for 18% errors. For example, the word "heterogeneous" interchangeably referred to nodule contents or echogenicity. While nodule dimensions and laterality were often described, US reports only described contents, echogenicity, vascularity, calcifications, borders, and lymphadenopathy, 46, 41, 17, 15, 9, and 41% of the time, respectively. Most nodule characteristics were equally likely to be described at hospital A compared with hospital B.

CONCLUSIONS

 NLP can automate extraction of critical information from thyroid US reports. However, ambiguous and incomplete reporting language hinders performance of NLP systems regardless of institutional setting. Standardized or synoptic thyroid US reports could improve NLP performance.

Machine learning and natural language processing to identify falls in electronic patient care records from ambulance attendances

We derived machine learning models utilizing features generated by natural language processing (NLP) of free-text data from an ambulance services provider to identify fall cases. The data comprised samples of electronic patient care records care records (ePCRs) from St John Western Australia (WA), the sole ambulance services provider in most of WA. We manually labeled fall cases by reviewing the free-text summary. The models used features including case characteristics (e.g., age) and text frequency-inverse document frequency (tf-idf) of each word of the free-text generated by NLP. Support vector machine (SVM) and random forest were used as classifiers. We compared the performance of the models against the manual identification of falls by recall, precision, and F-measure. A total of 9,447 cases (1%) were randomly sampled, of which 1,648 (17%) were labeled as fall. The best model was an SVM model using case characteristics and tf-idf's of the first 100 words of free-text, with recall of 0.84, precision of 0.86, and F-measure of 0.85. This performance was better than an SVM model with only case characteristics. Machine-learning models incorporated with features generated by NLP improved the performance of classifying fall cases compared with models without such features. Scope remains for further improvement.

Thyroid Ultrasound Reports: Will the Thyroid Imaging, Reporting, and Data System Improve Natural Language Processing Capture of Critical Thyroid Nodule Features?

BACKGROUND

Critical thyroid nodule features are contained in unstructured ultrasound (US) reports. The Thyroid Imaging, Reporting, and Data System (TI-RADS) uses five key features to risk stratify nodules and recommend appropriate intervention. This study aims to analyze the quality of US reporting and the potential benefit of Natural Language Processing (NLP) systems in efficiently capturing TI-RADS features from text reports.

MATERIALS AND METHOD

This retrospective study used free-text thyroid US reports from an academic center (A) and community hospital (B). Physicians created "gold standard" annotations by manually extracting TI-RADS features and clinical recommendations from reports to determine how often they were included. Similar annotations were created using an automated NLP system and compared with the gold standard.

RESULTS

Two hundred eighty-two reports contained 409 nodules at least 1-cm in maximum diameter. The gold standard identified three nodules (0.7%) which contained enough information to calculate a complete TI-RADS score. Shape was described most often (92.7% of nodules), whereas margins were described least often (11%). A median number of two TI-RADS features are reported per nodule. The NLP system was significantly less accurate than the gold standard in capturing echogenicity (27.5%) and margins (58.9%). One hundred eight nodule reports (26.4%) included clinical management recommendations, which were included more often at site A than B (33.9 versus 17%, P < 0.05).

CONCLUSIONS

These results suggest a gap between current US reporting styles and those needed to implement TI-RADS and achieve NLP accuracy. Synoptic reporting should prompt more complete thyroid US reporting, improved recommendations for intervention, and better NLP performance.