Use of natural language processing to improve predictive models for imaging utilization in children presenting to the emergency department

Expenditure mapping of pediatric imaging costs using a resource utilization band analysis of claims data

OBJECTIVE

To segregate imaging expenditures from claims data by resource utilization bands (RUBs) and underlying conditions to create an "expenditure map" of pediatric imaging costs.

METHODS

A Claims data for children enrolled in a commercial value-based plan were categorized by RUB 0 non-user, 1 healthy user, 2 low morbidity, 3 moderate morbidity, 4 high morbidity, & 5 very high morbidity. The per member per year (PMPY) expense, total imaging spend, and imaging modality with the highest spend were assessed for each RUB. Diagnosis categories associated with high imaging costs were also evaluated.

RESULTS

There were 40,022 pediatric plan members. 14% had imaging-related claims accounting for approximately $2.8 million in expenditures. Member distribution and mean PMPY expenditure RUB was respectively: RUB 0 (3,037, $0), RUB 1 (6,604, $7), RUB 2 - 13,698, $27), RUB 3 - 13,341, $87), RUB 4 (2,810, $268), RUB 5 (532, $841). RUB 3 had the largest total imaging costs at $1,159,523. The imaging modality with the greatest mean PMPY expense varied by RUB with radiography highest in lower RUBs and MRI highest in higher RUBs. The top 3 diagnoses associated with the highest total imaging costs were developmental disorders ($443,980), asthma ($388,797), and congenital heart disease ($294,977) and greatest mean PMPY imaging expenditures malignancy/leukemia ($3,100), transplant ($2,639), and tracheostomy ($1,661).

DISCUSSION

Expense mapping using claims data allows for a better understanding of the distribution of imaging costs across a covered pediatric population. This tool may assist organizations in planning effective cost-reduction initiatives and learning how imaging utilization varies by patient complexity in their system.

Development and validation of a machine learning model for prediction of comorbid major depression disorder among narcolepsy type 1

BACKGROUND

Major depression disorder (MDD) forms a common psychiatric comorbidity among patients with narcolepsy type 1 (NT1), yet its impact on patients with NT1 is often overlooked by neurologists. Currently, there is a lack of effective methods for accurately predicting MDD in patients with NT1.

OBJECTIVE

This study utilized machine learning (ML) algorithms to identify critical variables and developed the prediction model for predicting MDD in patients with NT1.

METHODS

The study included 267 NT1 patients from four sleep centers. The diagnosis of comorbid MDD was based on Diagnostic and Statistical Manual of Mental Disorders fifth edition (DSM-5). ML models, including six full models and six compact models, were developed using a training set. The performance of these models was compared in the testing set, and the optimal model was evaluated in the testing set. Various evaluation metrics, such as Area under the receiver operating curve (AUC), precision-recall (PR) curve and calibration curve were employed to assess and compare the performance of the ML models. Model interpretability was demonstrated using SHAP.

RESULT

In the testing set, the logistic regression (LG) model demonstrated superior performance compared to other ML models based on evaluation metrics such as AUC, PR curve, and calibration curve. The top eight features used in the LG model, ranked by feature importance, included social impact scale (SIS) score, narcolepsy severity scale (NSS) score, total sleep time, body mass index (BMI), education years, age of onset, sleep efficiency, sleep latency.

CONCLUSION

The study yielded a straightforward and practical ML model for the early identification of MDD in patients with NT1. A web-based tool for clinical applications was developed, which deserves further verification in diverse clinical settings.

Machine learning-Based model for prediction of Narcolepsy Type 1 in Patients with Obstructive Sleep Apnea with Excessive Daytime Sleepiness

Background

Excessive daytime sleepiness (EDS) forms a prevalent symptom of obstructive sleep apnea (OSA) and narcolepsy type 1 (NT1), while the latter might always be overlooked. Machine learning (ML) models can enable the early detection of these conditions, which has never been applied for diagnosis of NT1.

Objective

The study aimed to develop ML prediction models to help non-sleep specialist clinicians identify high probability of comorbid NT1 in patients with OSA early.

Methods

Totally, clinical features of 246 patients with OSA in three sleep centers were collected and analyzed for the development of nine ML models. LASSO regression was used for feature selection. Various metrics such as the area under the receiver operating curve (AUC), calibration curve, and decision curve analysis (DCA) were employed to evaluate and compare the performance of these ML models. Model interpretability was demonstrated by Shapley Additive explanations (SHAP).

Results

Based on the analysis of AUC, DCA, and calibration curves, the Gradient Boosting Machine (GBM) model demonstrated superior performance compared to other machine learning (ML) models. The top five features used in the GBM model, ranked by feature importance, were age of onset, total limb movements index, sleep latency, non-REM (Rapid Eye Movement) sleep stage 2 and severity of OSA.

Conclusion

The study yielded a simple and feasible screening ML-based model for the early identification of NT1 in patients with OSA, which warrants further verification in more extensive clinical practices.

Applications of natural language processing at emergency department triage: A narrative review

INTRODUCTION

Natural language processing (NLP) uses various computational methods to analyse and understand human language, and has been applied to data acquired at Emergency Department (ED) triage to predict various outcomes. The objective of this scoping review is to evaluate how NLP has been applied to data acquired at ED triage, assess if NLP based models outperform humans or current risk stratification techniques when predicting outcomes, and assess if incorporating free-text improve predictive performance of models when compared to predictive models that use only structured data.

METHODS

All English language peer-reviewed research that applied an NLP technique to free-text obtained at ED triage was eligible for inclusion. We excluded studies focusing solely on disease surveillance, and studies that used information obtained after triage. We searched the electronic databases MEDLINE, Embase, Cochrane Database of Systematic Reviews, Web of Science, and Scopus for medical subject headings and text keywords related to NLP and triage. Databases were last searched on 01/01/2022. Risk of bias in studies was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). Due to the high level of heterogeneity between studies and high risk of bias, a metanalysis was not conducted. Instead, a narrative synthesis is provided.

RESULTS

In total, 3730 studies were screened, and 20 studies were included. The population size varied greatly between studies ranging from 1.8 million patients to 598 triage notes. The most common outcomes assessed were prediction of triage score, prediction of admission, and prediction of critical illness. NLP models achieved high accuracy in predicting need for admission, triage score, critical illness, and mapping free-text chief complaints to structured fields. Incorporating both structured data and free-text data improved results when compared to models that used only structured data. However, the majority of studies (80%) were assessed to have a high risk of bias, and only one study reported the deployment of an NLP model into clinical practice.

CONCLUSION

Unstructured free-text triage notes have been used by NLP models to predict clinically relevant outcomes. However, the majority of studies have a high risk of bias, most research is retrospective, and there are few examples of implementation into clinical practice. Future work is needed to prospectively assess if applying NLP to data acquired at ED triage improves ED outcomes when compared to usual clinical practice.

Natural Language Processing of Nursing Notes: An Integrative Review

Natural language processing includes a variety of techniques that help to extract meaning from narrative data. In healthcare, medical natural language processing has been a growing field of study; however, little is known about its use in nursing. We searched PubMed, EMBASE, and CINAHL and found 689 studies, narrowed to 43 eligible studies using natural language processing in nursing notes. Data related to the study purpose, patient population, methodology, performance evaluation metrics, and quality indicators were extracted for each study. The majority (86%) of the studies were conducted from 2015 to 2021. Most of the studies (58%) used inpatient data. One of four studies used data from open-source databases. The most common standard terminologies used were the Unified Medical Language System and Systematized Nomenclature of Medicine, whereas nursing-specific standard terminologies were used only in eight studies. Full system performance metrics (eg, F score) were reported for 61% of applicable studies. The overall number of nursing natural language processing publications remains relatively small compared with the other medical literature. Future studies should evaluate and report appropriate performance metrics and use existing standard nursing terminologies to enable future scalability of the methods and findings.

Using machine learning approach to predict depression and anxiety among patients with epilepsy in China: A cross-sectional study

BACKGROUND

Anxiety and depression are the most prevalent comorbidities among epilepsy patients. The screen and diagnosis of anxiety and depression are quite important for the management of patients with epilepsy. In that case, the method for accurately predicting anxiety and depression needs to be further explored.

METHODS

A total of 480 patients with epilepsy (PWE) were enrolled in our study. Anxiety and Depressive symptoms were evaluated. Six machine learning models were used to predict anxiety and depression in patients with epilepsy. Receiver operating curve (ROC), decision curve analysis (DCA) and moDel Agnostic Language for Exploration and eXplanation (DALEX) package were used to evaluate the accuracy of machine learning models.

RESULTS

For anxiety, the area under the ROC curve was not significantly different between models. DCA revealed that random forest and multilayer perceptron has the largest net benefit within different probability threshold. DALEX revealed that random forest and multilayer perceptron were models with best performance and stigma had the highest feature importance. For depression, the results were much the same.

CONCLUSIONS

Methods created in this study may offer much help identifying PWE with high risk of anxiety and depression. The decision support system may be valuable for the everyday management of PWE. Further study is needed to test the outcome of applying this system to clinical settings.

Rational Usage of Fracture Imaging in Children and Adolescents

In this paper, authors introduce the basic prerequisite for rational, targeted, and above all, child-oriented diagnosis of fractures and dislocations in children and adolescents is in-depth prior knowledge of the special features of trauma in the growth age group. This review summarizes the authors' many years of experience and the state of the current pediatric traumatology literature. It aims to provide recommendations for rational, child-specific diagnostics appropriate to the child, especially for the area of extremity injuries in the growth age. The plain radiograph remains the indispensable standard in diagnosing fractures and dislocations of the musculoskeletal system in childhood and adolescence. Plain radiographs in two planes are the norm, but in certain situations, one plane is sufficient. X-rays of the opposite side in acute diagnostics are obsolete. Images to show consolidation after conservative treatment is rarely necessary. Before metal removal, however, they are indispensable. The upcoming diagnostical tool in pediatric trauma is ultrasound. More and more studies show that in elected injuries and using standardized protocols, fracture ultrasound is as accurate as plain radiographs to detect and control osseous and articular injuries. In acute trauma, CT scans have only a few indications, especially in epiphyseal fractures in adolescents, such as transitional fractures of the distal tibia or coronal shear fractures of the distal humerus. CT protocols must be adapted to children and adolescents to minimize radiation exposure. MRI has no indication in the detection or understanding of acute fractures in infants and children. It has its place in articular injuries of the knee and shoulder to show damage to ligaments, cartilage, and other soft tissues. Furthermore, MRI is useful in cases of remaining pain after trauma without radiological proof of a fracture and in the visualization of premature closure of growth plates after trauma to plan therapy. Several everyday examples of rational diagnostic workflows, as the authors recommend them, are mentioned. The necessity of radiation protection must be taken into consideration.

The Use and Structure of Emergency Nurses' Triage Narrative Data: Scoping Review

BACKGROUND

Emergency departments use triage to ensure that patients with the highest level of acuity receive care quickly and safely. Triage is typically a nursing process that is documented as structured and unstructured (free text) data. Free-text triage narratives have been studied for specific conditions but never reviewed in a comprehensive manner.

OBJECTIVE

The objective of this paper was to identify and map the academic literature that examines triage narratives. The paper described the types of research conducted, identified gaps in the research, and determined where additional review may be warranted.

METHODS

We conducted a scoping review of unstructured triage narratives. We mapped the literature, described the use of triage narrative data, examined the information available on the form and structure of narratives, highlighted similarities among publications, and identified opportunities for future research.

RESULTS

We screened 18,074 studies published between 1990 and 2022 in CINAHL, MEDLINE, Embase, Cochrane, and ProQuest Central. We identified 0.53% (96/18,074) of studies that directly examined the use of triage nurses' narratives. More than 12 million visits were made to 2438 emergency departments included in the review. In total, 82% (79/96) of these studies were conducted in the United States (43/96, 45%), Australia (31/96, 32%), or Canada (5/96, 5%). Triage narratives were used for research and case identification, as input variables for predictive modeling, and for quality improvement. Overall, 31% (30/96) of the studies offered a description of the triage narrative, including a list of the keywords used (27/96, 28%) or more fulsome descriptions (such as word counts, character counts, abbreviation, etc; 7/96, 7%). We found limited use of reporting guidelines (8/96, 8%).

CONCLUSIONS

The breadth of the identified studies suggests that there is widespread routine collection and research use of triage narrative data. Despite the use of triage narratives as a source of data in studies, the narratives and nurses who generate them are poorly described in the literature, and data reporting is inconsistent. Additional research is needed to describe the structure of triage narratives, determine the best use of triage narratives, and improve the consistent use of triage-specific data reporting guidelines.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2021-055132.

Individualized prediction of chronic kidney disease for the elderly in longevity areas in China: Machine learning approaches

Background

Chronic kidney disease (CKD) has become a major public health problem worldwide and has caused a huge social and economic burden, especially in developing countries. No previous study has used machine learning (ML) methods combined with longitudinal data to predict the risk of CKD development in 2 years amongst the elderly in China.

Methods

This study was based on the panel data of 925 elderly individuals in the 2012 baseline survey and 2014 follow-up survey of the Healthy Aging and Biomarkers Cohort Study (HABCS) database. Six ML models, logistic regression (LR), lasso regression, random forests (RF), gradient-boosted decision tree (GBDT), support vector machine (SVM), and deep neural network (DNN), were developed to predict the probability of CKD amongst the elderly in 2 years (the year of 2014). The decision curve analysis (DCA) provided a range of threshold probability of the outcome and the net benefit of each ML model.

Results

Amongst the 925 elderly in the HABCS 2014 survey, 289 (18.8%) had CKD. Compared with the other models, LR, lasso regression, RF, GBDT, and DNN had no statistical significance of the area under the receiver operating curve (AUC) value (>0.7), and SVM exhibited the lowest predictive performance (AUC = 0.633, p-value = 0.057). DNN had the highest positive predictive value (PPV) (0.328), whereas LR had the lowest (0.287). DCA results indicated that within the threshold ranges of ~0-0.03 and 0.37-0.40, the net benefit of GBDT was the largest. Within the threshold ranges of ~0.03-0.10 and 0.26-0.30, the net benefit of RF was the largest. Age was the most important predictor variable in the RF and GBDT models. Blood urea nitrogen, serum albumin, uric acid, body mass index (BMI), marital status, activities of daily living (ADL)/instrumental activities of daily living (IADL) and gender were crucial in predicting CKD in the elderly.

Conclusion

The ML model could successfully capture the linear and nonlinear relationships of risk factors for CKD in the elderly. The decision support system based on the predictive model in this research can help medical staff detect and intervene in the health of the elderly early.

Factors associated with poor self-management documented in home health care narrative notes for patients with heart failure

BACKGROUND

Patients with heart failure (HF) who actively engage in their own self-management have better outcomes. Extracting data through natural language processing (NLP) holds great promise for identifying patients with or at risk of poor self-management.

OBJECTIVE

To identify home health care (HHC) patients with HF who have poor self-management using NLP of narrative notes, and to examine patient factors associated with poor self-management.

METHODS

An NLP algorithm was applied to extract poor self-management documentation using 353,718 HHC narrative notes of 9,710 patients with HF. Sociodemographic and structured clinical data were incorporated into multivariate logistic regression models to identify factors associated with poor self-management.

RESULTS

There were 758 (7.8%) patients in this sample identified as having notes with language describing poor HF self-management. Younger age (OR 0.982, 95% CI 0.976-0.987, p < .001), longer length of stay in HHC (OR 1.036, 95% CI 1.029- 1.043, p < .001), diagnosis of diabetes (OR 1.47, 95% CI 1.3-1.67, p < .001) and depression (OR 1.36, 95% CI 1.09-1.68, p < .01), impaired decision-making (OR 1.64, 95% CI 1.37-1.95, p < .001), smoking (OR 1.7, 95% CI 1.4-2.04, p < .001), and shortness of breath with exertion (OR 1.25, 95% CI 1.1-1.42, p < .01) were associated with poor self-management.

CONCLUSIONS

Patients with HF who have poor self-management can be identified from the narrative notes in HHC using novel NLP methods. Meaningful information about the self-management of patients with HF can support HHC clinicians in developing individualized care plans to improve self-management and clinical outcomes.

Prediction of acute appendicitis among patients with undifferentiated abdominal pain at emergency department

Background

Early screening and accurately identifying Acute Appendicitis (AA) among patients with undifferentiated symptoms associated with appendicitis during their emergency visit will improve patient safety and health care quality. The aim of the study was to compare models that predict AA among patients with undifferentiated symptoms at emergency visits using both structured data and free-text data from a national survey.

Methods

We performed a secondary data analysis on the 2005-2017 United States National Hospital Ambulatory Medical Care Survey (NHAMCS) data to estimate the association between emergency department (ED) patients with the diagnosis of AA, and the demographic and clinical factors present at ED visits during a patient’s ED stay. We used binary logistic regression (LR) and random forest (RF) models incorporating natural language processing (NLP) to predict AA diagnosis among patients with undifferentiated symptoms.

Results

Among the 40,441 ED patients with assigned International Classification of Diseases (ICD) codes of AA and appendicitis-related symptoms between 2005 and 2017, 655 adults (2.3%) and 256 children (2.2%) had AA. For the LR model identifying AA diagnosis among adult ED patients, the c-statistic was 0.72 (95% CI: 0.69–0.75) for structured variables only, 0.72 (95% CI: 0.69–0.75) for unstructured variables only, and 0.78 (95% CI: 0.76–0.80) when including both structured and unstructured variables. For the LR model identifying AA diagnosis among pediatric ED patients, the c-statistic was 0.84 (95% CI: 0.79–0.89) for including structured variables only, 0.78 (95% CI: 0.72–0.84) for unstructured variables, and 0.87 (95% CI: 0.83–0.91) when including both structured and unstructured variables. The RF method showed similar c-statistic to the corresponding LR model.

Conclusions

We developed predictive models that can predict the AA diagnosis for adult and pediatric ED patients, and the predictive accuracy was improved with the inclusion of NLP elements and approaches.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01490-9.

Racial and Ethnic Differences in Emergency Department Diagnostic Imaging at US Children's Hospitals, 2016-2019

IMPORTANCE

Diagnostic imaging is frequently performed as part of the emergency department (ED) evaluation of children. Whether imaging patterns differ by race and ethnicity is unknown.

OBJECTIVE

To evaluate racial and ethnic differences in the performance of common ED imaging studies and to examine patterns across diagnoses.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study evaluated visits by patients younger than 18 years to 44 US children's hospital EDs from January 1, 2016, through December 31, 2019.

EXPOSURES

Non-Hispanic Black and Hispanic compared with non-Hispanic White race/ethnicity.

MAIN OUTCOMES AND MEASURES

The primary outcome was the proportion of visits for each race/ethnicity group with at least 1 diagnostic imaging study, defined as plain radiography, computed tomography, ultrasonography, and magnetic resonance imaging. The major diagnostic categories classification system was used to examine race/ethnicity differences in imaging rates by diagnoses.

RESULTS

A total of 13 087 522 visits by 6 230 911 children and adolescents (mean [SD] age, 5.8 [5.2] years; 52.7% male) occurred during the study period. Diagnostic imaging was performed during 3 689 163 visits (28.2%). Imaging was performed in 33.5% of visits by non-Hispanic White patients compared with 24.1% of visits by non-Hispanic Black patients (odds ratio [OR], 0.60; 95% CI, 0.60-0.60) and 26.1% of visits by Hispanic patients (OR, 0.66; 95% CI, 0.66-0.67). Adjusting for confounders, visits by non-Hispanic Black (adjusted OR, 0.82; 95% CI, 0.82-0.83) and Hispanic (adjusted OR, 0.87; 95% CI, 0.87-0.87) patients were less likely to include any imaging study compared with visits by non-Hispanic White patients. Limiting the analysis to only visits by nonhospitalized patients, the adjusted OR for imaging was 0.79 (95% CI, 0.79-0.80) for visits by non-Hispanic Black patients and 0.84 (95% CI, 0.84-0.85) for visits by Hispanic patients. Results were consistent in analyses stratified by public and private insurance groups and did not materially differ by diagnostic category.

CONCLUSIONS AND RELEVANCE

In this study, non-Hispanic Black and Hispanic children were less likely to receive diagnostic imaging during ED visits compared with non-Hispanic White children. Further investigation is needed to understand and mitigate these potential disparities in health care delivery and to evaluate the effect of these differential imaging patterns on patient outcomes.