Validation of Prediction Models for Critical Care Outcomes Using Natural Language Processing of Electronic Health Record Data

Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study

Background

Prediction models have demonstrated a range of applications across medicine, including using electronic health record (EHR) data to identify hospital readmission and mortality risk. Large language models (LLMs) can transform unstructured EHR text into structured features, which can then be integrated into statistical prediction models, ensuring that the results are both clinically meaningful and interpretable.

Objective

This study aims to compare the classification decisions made by clinical experts with those generated by a state-of-the-art LLM, using terms extracted from a large EHR data set of individuals with mental health disorders seen in emergency departments (EDs).

Methods

Using a dataset from the EHR systems of more than 50 health care provider organizations in the United States from 2016 to 2021, we extracted all clinical terms that appeared in at least 1000 records of individuals admitted to the ED for a mental health-related problem from a source population of over 6 million ED episodes. Two experienced mental health clinicians (one medically trained psychiatrist and one clinical psychologist) reached consensus on the classification of EHR terms and diagnostic codes into categories. We evaluated an LLM's agreement with clinical judgment across three classification tasks as follows: (1) classify terms into "mental health" or "physical health", (2) classify mental health terms into 1 of 42 prespecified categories, and (3) classify physical health terms into 1 of 19 prespecified broad categories.

Results

There was high agreement between the LLM and clinical experts when categorizing 4553 terms as "mental health" or "physical health" (κ=0.77, 95% CI 0.75-0.80). However, there was still considerable variability in LLM-clinician agreement on the classification of mental health terms (κ=0.62, 95% CI 0.59-0.66) and physical health terms (κ=0.69, 95% CI 0.67-0.70).

Conclusions

The LLM displayed high agreement with clinical experts when classifying EHR terms into certain mental health or physical health term categories. However, agreement with clinical experts varied considerably within both sets of mental and physical health term categories. Importantly, the use of LLMs presents an alternative to manual human coding, presenting great potential to create interpretable features for prediction models.

Comparison of six natural language processing approaches to assessing firearm access in Veterans Health Administration electronic health records

OBJECTIVE

Access to firearms is associated with increased suicide risk. Our aim was to develop a natural language processing approach to characterizing firearm access in clinical records.

MATERIALS AND METHODS

We used clinical notes from 36 685 Veterans Health Administration (VHA) patients between April 10, 2023 and April 10, 2024. We expanded preexisting firearm term sets using subject matter experts and generated 250-character snippets around each firearm term appearing in notes. Annotators labeled 3000 snippets into three classes. Using these annotated snippets, we compared four nonneural machine learning models (random forest, bagging, gradient boosting, logistic regression with ridge penalization) and two versions of Bidirectional Encoder Representations from Transformers, or BERT (specifically, BioBERT and Bio-ClinicalBERT) for classifying firearm access as "definite access", "definitely no access", or "other".

RESULTS

Firearm terms were identified in 36 685 patient records (41.3%), 33.7% of snippets were categorized as definite access, 9.0% as definitely no access, and 57.2% as "other". Among models classifying firearm access, five of six had acceptable performance, with BioBERT and Bio-ClinicalBERT performing best, with F1s of 0.876 (95% confidence interval, 0.874-0.879) and 0.896 (95% confidence interval, 0.894-0.899), respectively.

DISCUSSION AND CONCLUSION

Firearm-related terminology is common in the clinical records of VHA patients. The ability to use text to identify and characterize patients' firearm access could enhance suicide prevention efforts, and five of our six models could be used to identify patients for clinical interventions.

Preliminary findings regarding the association between patient demographics and ED experience scores across a regional health system: A cross sectional study using natural language processing of patient comments

OBJECTIVE

Existing literature shows associations between patient demographics and reported experiences of care, but this relationship is poorly understood. Our objective was to use natural language processing of patient comments to gain insight into associations between patient demographics and experiences of care.

METHODS

This is a cross-sectional study of 14,848 unique emergency department (ED) patient visits from 1/1/2020 to 12/31/2020. Patients discharged from one of 16 ED sites in a regional health system who filled out a patient experience survey with comments were included. This study had two outcome variables: (1) positive vs. non-positive (negative/neutral) comment sentiment, and (2) promoter vs. non-promoter status (based on NRCHealth's Net Promoter Score; likelihood to recommend of 9 or 10 are considered "promoters", while scores of 8 or below are "non-promoters"). We used natural language processing to sort patient comments into topics and sentiments. Logistic regression with mediation analysis was used to estimate the associations between patient demographics and the following: (1) comments about compassion vs. other topics, (2) positive comments, and (3) patient experience, defined as likelihood to recommend.

RESULTS

Comments about care and compassion (51 % of total comments) had highly positive sentiment (97 %), compared to mixed sentiment for other topics. Older, male, and Asian patients were more likely to comment on compassion and most likely to make positive comments. Our mediation analysis suggests that the demographic association with positive patient comments and net promoter scores was mediated by their focus on care and compassion as a primary comment theme for their visit. Notably, the overall percentage of patients providing comments was only 1.8 %, raising concerns about whether data currently used for hospital and physician feedback has adequate validity to yield meaningful insights.

CONCLUSIONS

The increased likelihood of specific patient sub-groups to comment on compassionate care may explain previously reported differences in experience by patient demographics.

Intensive care unit nurses' experiences of nursing concerns, activities, and documentation on patient deterioration: A focus-group study

BACKGROUND

Although prognosis prediction models using nursing documentation have good predictive performance, the experiences of intensive care unit nurses related to nursing activities and documentation when a patient's condition deteriorates are yet to be explored.

OBJECTIVE

The aim of this study was to explore nurses' experiences of nursing activities and documentation in intensive care units when a patient's condition deteriorates.

METHODS

This was a descriptive qualitative study using focus-group interviews with intensive care unit nurses in tertiary or university-affiliated hospitals. In total, 19 registered nurses with at least 1 year of clinical experience in the adult intensive care unit were recruited using a purposive sampling method. Five focus-group interviews were conducted, and the data were analysed through a qualitative content analysis.

RESULTS

Intensive care unit nurses' experiences with patient deterioration were classified into four main categories-perceived patient deterioration; endeavours to verify nurses' concerns; nursing activities to improve a patient's condition; and optimising documentation practices-which comprised 12 subcategories. Intensive care unit nurses recognise patient deterioration through nursing activities and documentation, and the two processes influence each other. However, nursing activities related to nurses' concerns were mainly handed over verbally rather than documented due to the inflexibility of the available standardised forms and the potential uncertainty of those concerns.

CONCLUSIONS

The findings reveal how intensive care unit nurses perceive, intervene, and document the condition of a deteriorating patient. Nurses' concerns may be the first sign of a patient's deteriorating condition and are therefore crucial for minimising patient risk. Therefore, efforts to systematically document nurses' concerns may contribute to improving patient outcomes.

Age modulates the predictive value of self-reported sleepiness for all-cause mortality risk: insights from a comprehensive national database of veterans

STUDY OBJECTIVES

Excessive daytime sleepiness is prevalent and overwhelmingly stems from disturbed sleep. We hypothesized that age modulates the association between excessive daytime sleepiness and increased all-cause mortality.

METHODS

We utilized the Veterans' Health Administration data from 1999-2022. We enrolled participants with sleep related International Classification of Diseases 9/10 codes or sleep services. A natural language processing pipeline was developed and validated to extract the Epworth Sleepiness Scale (ESS) as a self-reported tool to measure excessive daytime sleepiness from physician progress notes. The natural language processing's accuracy was assessed through manual annotation of 470 notes. Participants were categorized into normal-ESS (ESS 0-10) and high-ESS (ESS 11-24). We created 3 age groups: < 50 years, 50 to < 65 years, and ≥ 65 years. The adjusted odds ratio of mortality was calculated for age, body mass index, sex, race, ethnicity, and the Charlson Comorbidity Index, using normal-ESS as the reference. Subsequently, we conducted age stratified analysis.

RESULTS

The first ESS records were extracted from 423,087 veterans with a mean age of 54.8 (± 14.6), mean body mass index of 32.6 (± 6.2), and 90.5% male. The adjusted odds ratio across all ages was 17% higher (1.15, 1.19) in the high-ESS category. The adjusted odds ratio s only became statistically significant for individuals aged ≥ 50 years in the high-ESS compared to the normal-ESS category (< 50 years: 1.02 [0.96, 1.08], 50 to < 65 years 1.13[1.10, 1.16]; ≥ 65 years: 1.25 [1.21, 1.28]).

CONCLUSIONS

High-ESS predicted increased mortality only in participants aged 50 and older. Further research is required to identify this differential behavior in relation to age.

CITATION

Maghsoudi A, Azarian M, Sharafkhaneh A, et al. Age modulates the predictive value of self-reported sleepiness for all-cause mortality risk: insights from a comprehensive national database of veterans. J Clin Sleep Med. 2024;20(11):1785-1792.

Enhancing readmission prediction models by integrating insights from home healthcare notes: Retrospective cohort study

BACKGROUND

Hospital readmission is an important indicator of inpatient care quality and a significant driver of increasing medical costs. Therefore, it is important to explore the effects of postdischarge information, particularly from home healthcare notes, on enhancing readmission prediction models. Despite the use of Natural Language Processing (NLP) and machine learning in prediction model development, current studies often overlook insights from home healthcare notes.

OBJECTIVE

This study aimed to develop prediction models for 30-day readmissions using home healthcare notes and structured data. In addition, it explored the development of 14- and 180-day prediction models using variables in the 30-day model.

DESIGN

A retrospective observational cohort study.

SETTING(S)

This study was conducted at Ajou University School of Medicine in South Korea.

PARTICIPANTS

Data from electronic health records, encompassing demographic characteristics of 1819 participants, along with information on conditions, drug, and home healthcare, were utilized.

METHODS

Two distinct models were developed for each prediction window (30-, 14-, 180-day): the traditional model, which utilized structured variables alone, and the common data model (CDM)-NLP model, which incorporated structured and topic variables extracted from home healthcare notes. BERTopic facilitated topic generation and risk probability, representing the likelihood of documents being assigned to specific topics. Feature selection involved experimenting with various algorithms. The best-performing algorithm, determined using the area under the receiver operating characteristic curve (AUROC), was used for model development. Model performance was assessed using various learning metrics including AUROC.

RESULTS

Among 1819 patients, 251 (13.80 %) experienced 30-day readmission. The least absolute shrinkage and selection operator was used for feature extraction and model development. The 15 structured features were used in the traditional model. Moreover, five additional topic variables from the home healthcare notes were applied in the CDM-NLP model. The AUROC of the traditional model was 0.739 (95 % CI: 0.672-0.807). The AUROC of the CDM-NLP model was high at 0.824 (95 % CI: 0.768-0.880), which indicated an outstanding performance. The topics in the CDM-NLP model included emotional distress, daily living functions, nutrition, postoperative status, and cardiorespiratory issues. In extended prediction model development for 14- and 180-day readmissions, the CDM-NLP consistently outperformed the traditional model.

CONCLUSIONS

This study developed effective prediction models using both structured and unstructured data, thereby emphasizing the significance of postdischarge information from home healthcare notes in readmission prediction.

Natural language processing in the intensive care unit: A scoping review

Objectives

Natural language processing (NLP) is a branch of artificial intelligence focused on enabling computers to interpret and analyse text-based data. The intensive care specialty is known to generate large volumes of data, including free-text, however, NLP applications are not commonly used either in critical care clinical research or quality improvement projects. This review aims to provide an overview of how NLP has been used in the intensive care specialty and promote an understanding of NLP's potential future clinical applications.

Design

Scoping review.

Data sources

A systematic search was developed with an information specialist and deployed on the PubMed electronic journal database. Results were restricted to the last 10 years to ensure currency.

Review methods

Screening and data extraction were undertaken by two independent reviewers, with any disagreements resolved by a third. Given the heterogeneity of the eligible articles, a narrative synthesis was conducted.

Results

Eighty-seven eligible articles were included in the review. The most common type (n = 24) were studies that used NLP-derived features to predict clinical outcomes, most commonly mortality (n = 16). Next were articles that used NLP to identify a specific concept (n = 23), including sepsis, family visitation and mental health disorders. Most studies only described the development and internal validation of their algorithm (n = 79), and only one reported the implementation of an algorithm in a clinical setting.

Conclusions

Natural language processing has been used for a variety of purposes in the ICU context. Increasing awareness of these techniques amongst clinicians may lead to more clinically relevant algorithms being developed and implemented.

A nursing note-aware deep neural network for predicting mortality risk after hospital discharge

BACKGROUND

ICU readmissions and post-discharge mortality pose significant challenges. Previous studies used EHRs and machine learning models, but mostly focused on structured data. Nursing records contain crucial unstructured information, but their utilization is challenging. Natural language processing (NLP) can extract structured features from clinical text. This study proposes the Crucial Nursing Description Extractor (CNDE) to predict post-ICU discharge mortality rates and identify high-risk patients for unplanned readmission by analyzing electronic nursing records.

OBJECTIVE

Developed a deep neural network (NurnaNet) with the ability to perceive nursing records, combined with a bio-clinical medicine pre-trained language model (BioClinicalBERT) to analyze the electronic health records (EHRs) in the MIMIC III dataset to predict the death of patients within six month and two year risk.

DESIGN

A cohort and system development design was used.

SETTING(S)

Based on data extracted from MIMIC-III, a database of critically ill in the US between 2001 and 2012, the results were analyzed.

PARTICIPANTS

We calculated patients' age using admission time and date of birth information from the MIMIC dataset. Patients under 18 or over 89 years old, or who died in the hospital, were excluded. We analyzed 16,973 nursing records from patients' ICU stays.

METHODS

We have developed a technology called the Crucial Nursing Description Extractor (CNDE), which extracts key content from text. We use the logarithmic likelihood ratio to extract keywords and combine BioClinicalBERT. We predict the survival of discharged patients after six months and two years and evaluate the performance of the model using precision, recall, the F1-score, the receiver operating characteristic curve (ROC curve), the area under the curve (AUC), and the precision-recall curve (PR curve).

RESULTS

The research findings indicate that NurnaNet achieved good F1-scores (0.67030, 0.70874) within six months and two years. Compared to using BioClinicalBERT alone, there was an improvement in performance of 2.05 % and 1.08 % for predictions within six months and two years, respectively.

CONCLUSIONS

CNDE can effectively reduce long-form records and extract key content. NurnaNet has a good F1-score in analyzing the data of nursing records, which helps to identify the risk of death of patients after leaving the hospital and adjust the regular follow-up and treatment plan of relevant medical care as soon as possible.

Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models

BACKGROUND

Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms.

OBJECTIVES

This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes.

METHODS

The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level.

RESULTS

A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model.

CONCLUSIONS

The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.

Efficient screening of pharmacological broad-spectrum anti-cancer peptides utilizing advanced bidirectional Encoder representation from Transformers strategy

In the vanguard of oncological advancement, this investigation delineates the integration of deep learning paradigms to refine the screening process for Anticancer Peptides (ACPs), epitomizing a new frontier in broad-spectrum oncolytic therapeutics renowned for their targeted antitumor efficacy and specificity. Conventional methodologies for ACP identification are marred by prohibitive time and financial exigencies, representing a formidable impediment to the evolution of precision oncology. In response, our research heralds the development of a groundbreaking screening apparatus that marries Natural Language Processing (NLP) with the Pseudo Amino Acid Composition (PseAAC) technique, thereby inaugurating a comprehensive ACP compendium for the extraction of quintessential primary and secondary structural attributes. This innovative methodological approach is augmented by an optimized BERT model, meticulously calibrated for ACP detection, which conspicuously surpasses existing BERT variants and traditional machine learning algorithms in both accuracy and selectivity. Subjected to rigorous validation via five-fold cross-validation and external assessment, our model exhibited exemplary performance, boasting an average Area Under the Curve (AUC) of 0.9726 and an F1 score of 0.9385, with external validation further affirming its prowess (AUC of 0.9848 and F1 of 0.9371). These findings vividly underscore the method's unparalleled efficacy and prospective utility in the precise identification and prognostication of ACPs, significantly ameliorating the financial and temporal burdens traditionally associated with ACP research and development. Ergo, this pioneering screening paradigm promises to catalyze the discovery and clinical application of ACPs, constituting a seminal stride towards the realization of more efficacious and economically viable precision oncology interventions.

Performance of a Large Language Model in the Generation of Clinical Guidelines for Antibiotic Prophylaxis in Spine Surgery

OBJECTIVE

Large language models, such as chat generative pre-trained transformer (ChatGPT), have great potential for streamlining medical processes and assisting physicians in clinical decision-making. This study aimed to assess the potential of ChatGPT's 2 models (GPT-3.5 and GPT-4.0) to support clinical decision-making by comparing its responses for antibiotic prophylaxis in spine surgery to accepted clinical guidelines.

METHODS

ChatGPT models were prompted with questions from the North American Spine Society (NASS) Evidence-based Clinical Guidelines for Multidisciplinary Spine Care for Antibiotic Prophylaxis in Spine Surgery (2013). Its responses were then compared and assessed for accuracy.

RESULTS

Of the 16 NASS guideline questions concerning antibiotic prophylaxis, 10 responses (62.5%) were accurate in ChatGPT's GPT-3.5 model and 13 (81%) were accurate in GPT-4.0. Twenty-five percent of GPT-3.5 answers were deemed as overly confident while 62.5% of GPT-4.0 answers directly used the NASS guideline as evidence for its response.

CONCLUSION

ChatGPT demonstrated an impressive ability to accurately answer clinical questions. GPT-3.5 model's performance was limited by its tendency to give overly confident responses and its inability to identify the most significant elements in its responses. GPT-4.0 model's responses had higher accuracy and cited the NASS guideline as direct evidence many times. While GPT-4.0 is still far from perfect, it has shown an exceptional ability to extract the most relevant research available compared to GPT-3.5. Thus, while ChatGPT has shown far-reaching potential, scrutiny should still be exercised regarding its clinical use at this time.

Heart rate complexity helps mortality prediction in the intensive care unit: A pilot study using artificial intelligence

BACKGROUND

In intensive care units (ICUs), accurate mortality prediction is crucial for effective patient management and resource allocation. The Simplified Acute Physiology Score II (SAPS-2), though commonly used, relies heavily on comprehensive clinical data and blood samples. This study sought to develop an artificial intelligence (AI) model utilizing key hemodynamic parameters to predict ICU mortality within the first 24 h and assess its performance relative to SAPS-2.

METHODS

We conducted an analysis of select hemodynamic parameters and the structure of heart rate curves to identify potential predictors of ICU mortality. A machine-learning model was subsequently trained and validated on distinct patient cohorts. The AI algorithm's performance was then compared to the SAPS-2, focusing on classification accuracy, calibration, and generalizability.

MEASUREMENTS AND MAIN RESULTS

The study included 1298 ICU admissions from March 27th, 2015, to March 27th, 2017. An additional cohort from 2022 to 2023 comprised 590 patients, resulting in a total dataset of 1888 patients. The observed mortality rate stood at 24.0%. Key determinants of mortality were the Glasgow Coma Scale score, heart rate complexity, patient age, duration of diastolic blood pressure below 50 mmHg, heart rate variability, and specific mean and systolic blood pressure thresholds. The AI model, informed by these determinants, exhibited a performance profile in predicting mortality that was comparable, if not superior, to the SAPS-2.

CONCLUSIONS

The AI model, which integrates heart rate and blood pressure curve analyses with basic clinical parameters, provides a methodological approach to predict in-hospital mortality in ICU patients. This model offers an alternative to existing tools that depend on extensive clinical data and laboratory inputs. Its potential integration into ICU monitoring systems may facilitate more streamlined mortality prediction processes.

The Growing Impact of Natural Language Processing in Healthcare and Public Health

Natural Language Processing (NLP) is a subset of Artificial Intelligence, specifically focused on understanding and generating human language. NLP technologies are becoming more prevalent in healthcare and hold potential solutions to current problems. Some examples of existing and future uses include: public sentiment analysis in relation to health policies, electronic health record (EHR) screening, use of speech to text technology for extracting EHR data from point of care, patient communications, accelerated identification of eligible clinical trial candidates through automated searches and access of health data to assist in informed treatment decisions. This narrative review aims to summarize the current uses of NLP in healthcare, highlight successful implementation of computational linguistics-based approaches, and identify gaps, limitations, and emerging trends within the subfield of NLP in public health. The online databases Google Scholar and PubMed were scanned for papers published between 2018 and 2023. Keywords "Natural Language Processing, Health Policy, Large Language Models" were utilized in the initial search. Then, papers were limited to those written in English. Each of the 27 selected papers was subject to careful analysis, and their relevance in relation to NLP and healthcare respectively is utilized in this review. NLP and deep learning technologies scan large datasets, extracting valuable insights in various realms. This is especially significant in healthcare where huge amounts of data exist in the form of unstructured text. Automating labor intensive and tedious tasks with language processing algorithms, using text analytics systems and machine learning to analyze social media data and extracting insights from unstructured data allows for better public sentiment analysis, enhancement of risk prediction models, improved patient communication, and informed treatment decisions. In the recent past, some studies have applied NLP tools to social media posts to evaluate public sentiment regarding COVID-19 vaccine use. Social media data also has the capacity to be harnessed to develop pandemic prediction models based on reported symptoms. Furthermore, NLP has the potential to enhance healthcare delivery across the globe. Advanced language processing techniques such as Speech Recognition (SR) and Natural Language Understanding (NLU) tools can help overcome linguistic barriers and facilitate efficient communication between patients and healthcare providers.

Impact of possible errors in natural language processing-derived data on downstream epidemiologic analysis

Objective

To assess the impact of potential errors in natural language processing (NLP) on the results of epidemiologic studies.

Materials and Methods

We utilized data from three outcomes research studies where the primary predictor variable was generated using NLP. For each of these studies, Monte Carlo simulations were applied to generate datasets simulating potential errors in NLP-derived variables. We subsequently fit the original regression models to these partially simulated datasets and compared the distribution of coefficient estimates to the original study results.

Results

Among the four models evaluated, the mean change in the point estimate of the relationship between the predictor variable and the outcome ranged from -21.9% to 4.12%. In three of the four models, significance of this relationship was not eliminated in a single of the 500 simulations, and in one model it was eliminated in 12% of simulations. Mean changes in the estimates for confounder variables ranged from 0.27% to 2.27% and significance of the relationship was eliminated between 0% and 9.25% of the time. No variables underwent a shift in the direction of its interpretation.

Discussion

Impact of simulated NLP errors on the results of epidemiologic studies was modest, with only small changes in effect estimates and no changes in the interpretation of the findings (direction and significance of association with the outcome) for either the NLP-generated variables or other variables in the models.

Conclusion

NLP errors are unlikely to affect the results of studies that use NLP as the source of data.

Risk Prediction Models for Hospital Mortality in General Medical Patients: A Systematic Review

Objective

To systematically review contemporary prediction models for hospital mortality developed or validated in general medical patients.

Methods

We screened articles in five databases, from January 1, 2010, through April 7, 2022, and the bibliography of articles selected for final inclusion. We assessed the quality for risk of bias and applicability using the Prediction Model Risk of Bias Assessment Tool (PROBAST) and extracted data using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist. Two investigators independently screened each article, assessed quality, and extracted data.

Results

From 20,424 unique articles, we identified 15 models in 8 studies across 10 countries. The studies included 280,793 general medical patients and 19,923 hospital deaths. Models included 7 early warning scores, 2 comorbidities indices, and 6 combination models. Ten models were studied in all general medical patients (general models) and 7 in general medical patients with infection (infection models). Of the 15 models, 13 were developed using logistic or Poisson regression and 2 using machine learning methods. Also, 4 of 15 models reported on handling of missing values. None of the infection models had high discrimination, whereas 4 of 10 general models had high discrimination (area under curve >0.8). Only 1 model appropriately assessed calibration. All models had high risk of bias; 4 of 10 general models and 5 of 7 infection models had low concern for applicability for general medical patients.

Conclusion

Mortality prediction models for general medical patients were sparse and differed in quality, applicability, and discrimination. These models require hospital-level validation and/or recalibration in general medical patients to guide mortality reduction interventions.

Development of a population-level prediction model for intensive care unit (ICU) survivorship and mortality in older adults: A population-based cohort study

Background and Aims

Given the growing utilization of critical care services by an aging population, development of population-level risk models which predict intensive care unit (ICU) survivorship and mortality may offer advantages for researchers and health systems. Our objective was to develop a risk model for ICU survivorship and mortality among community dwelling older adults.

Methods

This was a population-based cohort study of 48,127 patients who were 50 years and older with at least one primary care visit between January 1, 2017, and December 31, 2017. We used electronic health record (EHR) data to identify variables predictive of ICU survivorship.

Results

ICU admission and mortality within 2 years after index primary care visit date were used to divide patients into three groups of "alive without ICU admission", "ICU survivors," and "death." Multinomial logistic regression was used to identify EHR predictive variables for the three patient outcomes. Cross-validation by randomly splitting the data into derivation and validation data sets (60:40 split) was used to identify predictor variables and validate model performance using area under the receiver operating characteristics (AUC) curve. In our overall sample, 92.2% of patients were alive without ICU admission, 6.2% were admitted to the ICU at least once and survived, and 1.6% died. Greater deciles of age over 50 years, diagnoses of chronic obstructive pulmonary disorder or chronic heart failure, and laboratory abnormalities in alkaline phosphatase, hematocrit, and albumin contributed highest risk score weights for mortality. Risk scores derived from the model discriminated between patients that died versus remained alive without ICU admission (AUC = 0.858), and between ICU survivors versus alive without ICU admission (AUC = 0.765).

Conclusion

Our risk scores provide a feasible and scalable tool for researchers and health systems to identify patient cohorts at increased risk for ICU admission and survivorship. Further studies are needed to prospectively validate the risk scores in other patient populations.

Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges

The rapid adoption of electronic health record (EHR) systems in US hospitals from 2008 to 2014 produced novel data elements for analysis. Concurrent innovations in computing architecture and machine learning (ML) algorithms have made rapid consumption of health data feasible and a powerful engine for clinical innovation. In critical care research, the net convergence of these trends has resulted in an exponential increase in outcome prediction research. In the following article, we explore the history of outcome prediction in the intensive care unit (ICU), the growing use of EHR data, and the rise of artificial intelligence and ML (AI) in critical care.

AD-BERT: Using pre-trained language model to predict the progression from mild cognitive impairment to Alzheimer's disease

OBJECTIVE

We develop a deep learning framework based on the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model using unstructured clinical notes from electronic health records (EHRs) to predict the risk of disease progression from Mild Cognitive Impairment (MCI) to Alzheimer's Disease (AD).

METHODS

We identified 3657 patients diagnosed with MCI together with their progress notes from Northwestern Medicine Enterprise Data Warehouse (NMEDW) between 2000 and 2020. The progress notes no later than the first MCI diagnosis were used for the prediction. We first preprocessed the notes by deidentification, cleaning and splitting into sections, and then pre-trained a BERT model for AD (named AD-BERT) based on the publicly available Bio+Clinical BERT on the preprocessed notes. All sections of a patient were embedded into a vector representation by AD-BERT and then combined by global MaxPooling and a fully connected network to compute the probability of MCI-to-AD progression. For validation, we conducted a similar set of experiments on 2563 MCI patients identified at Weill Cornell Medicine (WCM) during the same timeframe.

RESULTS

Compared with the 7 baseline models, the AD-BERT model achieved the best performance on both datasets, with Area Under receiver operating characteristic Curve (AUC) of 0.849 and F1 score of 0.440 on NMEDW dataset, and AUC of 0.883 and F1 score of 0.680 on WCM dataset.

CONCLUSION

The use of EHRs for AD-related research is promising, and AD-BERT shows superior predictive performance in modeling MCI-to-AD progression prediction. Our study demonstrates the utility of pre-trained language models and clinical notes in predicting MCI-to-AD progression, which could have important implications for improving early detection and intervention for AD.

Towards Medical Billing Automation: NLP for Outpatient Clinician Note Classification

Objectives

Our primary objective was to develop a natural language processing approach that accurately predicts outpatient Evaluation and Management (E/M) level of service (LoS) codes using clinicians' notes from a health system electronic health record. A secondary objective was to investigate the impact of clinic note de-identification on document classification performance.

Methods

We used retrospective outpatient office clinic notes from four medical and surgical specialties. Classification models were fine-tuned on the clinic notes datasets and stratified by subspecialty. The success criteria for the classification tasks were the classification accuracy and F1-scores on internal test data. For the secondary objective, the dataset was de-identified using Named Entity Recognition (NER) to remove protected health information (PHI), and models were retrained.

Results

The models demonstrated similar predictive performance across different specialties, except for internal medicine, which had the lowest classification accuracy across all model architectures. The models trained on the entire note corpus achieved an E/M LoS CPT code classification accuracy of 74.8% (CI 95: 74.1-75.6). However, the de-identified note corpus showed a markedly lower classification accuracy of 48.2% (CI 95: 47.7-48.6) compared to the model trained on the identified notes.

Conclusion

The study demonstrates the potential of NLP-based document classifiers to accurately predict E/M LoS CPT codes using clinical notes from various medical and procedural specialties. The models' performance suggests that the classification task's complexity merits further investigation. The de-identification experiment demonstrated that de-identification may negatively impact classifier performance. Further research is needed to validate the performance of our NLP classifiers in different healthcare settings and patient populations and to investigate the potential implications of de-identification on model performance.

A Bayesian approach to predictive uncertainty in chemotherapy patients at risk of acute care utilization

BACKGROUND

Machine learning (ML) predictions are becoming increasingly integrated into medical practice. One commonly used method, ℓ1-penalised logistic regression (LASSO), can estimate patient risk for disease outcomes but is limited by only providing point estimates. Instead, Bayesian logistic LASSO regression (BLLR) models provide distributions for risk predictions, giving clinicians a better understanding of predictive uncertainty, but they are not commonly implemented.

METHODS

This study evaluates the predictive performance of different BLLRs compared to standard logistic LASSO regression, using real-world, high-dimensional, structured electronic health record (EHR) data from cancer patients initiating chemotherapy at a comprehensive cancer centre. Multiple BLLR models were compared against a LASSO model using an 80-20 random split using 10-fold cross-validation to predict the risk of acute care utilization (ACU) after starting chemotherapy.

FINDINGS

This study included 8439 patients. The LASSO model predicted ACU with an area under the receiver operating characteristic curve (AUROC) of 0.806 (95% CI: 0.775-0.834). BLLR with a Horseshoe+ prior and a posterior approximated by Metropolis-Hastings sampling showed similar performance: 0.807 (95% CI: 0.780-0.834) and offers the advantage of uncertainty estimation for each prediction. In addition, BLLR could identify predictions too uncertain to be automatically classified. BLLR uncertainties were stratified by different patient subgroups, demonstrating that predictive uncertainties significantly differ across race, cancer type, and stage.

INTERPRETATION

BLLRs are a promising yet underutilised tool that increases explainability by providing risk estimates while offering a similar level of performance to standard LASSO-based models. Additionally, these models can identify patient subgroups with higher uncertainty, which can augment clinical decision-making.

FUNDING

This work was supported in part by the National Library Of Medicine of the National Institutes of Health under Award Number R01LM013362. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Natural Language Processing in a Clinical Decision Support System for the Identification of Venous Thromboembolism: Algorithm Development and Validation

BACKGROUND

It remains unknown whether capturing data from electronic health records (EHRs) using natural language processing (NLP) can improve venous thromboembolism (VTE) detection in different clinical settings.

OBJECTIVE

The aim of this study was to validate the NLP algorithm in a clinical decision support system for VTE risk assessment and integrated care (DeVTEcare) to identify VTEs from EHRs.

METHODS

All inpatients aged ≥18 years in the Sixth Medical Center of the Chinese People's Liberation Army General Hospital from January 1 to December 31, 2021, were included as the validation cohort. The sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR-, respectively), area under the receiver operating characteristic curve (AUC), and F1-scores along with their 95% CIs were used to analyze the performance of the NLP tool, with manual review of medical records as the reference standard for detecting deep vein thrombosis (DVT) and pulmonary embolism (PE). The primary end point was the performance of the NLP approach embedded into the EHR for VTE identification. The secondary end points were the performances to identify VTE among different hospital departments with different VTE risks. Subgroup analyses were performed among age, sex, and the study season.

RESULTS

Among 30,152 patients (median age 56 [IQR 41-67] years; 14,247/30,152, 47.3% females), the prevalence of VTE, PE, and DVT was 2.1% (626/30,152), 0.6% (177/30,152), and 1.8% (532/30,152), respectively. The sensitivity, specificity, LR+, LR-, AUC, and F1-score of NLP-facilitated VTE detection were 89.9% (95% CI 87.3%-92.2%), 99.8% (95% CI 99.8%-99.9%), 483 (95% CI 370-629), 0.10 (95% CI 0.08-0.13), 0.95 (95% CI 0.94-0.96), and 0.90 (95% CI 0.90-0.91), respectively. Among departments of surgery, internal medicine, and intensive care units, the highest specificity (100% vs 99.7% vs 98.8%, respectively), LR+ (3202 vs 321 vs 77, respectively), and F1-score (0.95 vs 0.89 vs 0.92, respectively) were in the surgery department (all P<.001). Among low, intermediate, and high VTE risks in hospital departments, the low-risk department had the highest AUC (1.00 vs 0.94 vs 0.96, respectively) and F1-score (0.97 vs 0.90 vs 0.90, respectively) as well as the lowest LR- (0.00 vs 0.13 vs 0.08, respectively) (DeLong test for AUC; all P<.001). Subgroup analysis of the age, sex, and season demonstrated consistently good performance of VTE detection with >87% sensitivity and specificity and >89% AUC and F1-score. The NLP algorithm performed better among patients aged ≤65 years than among those aged >65 years (F1-score 0.93 vs 0.89, respectively; P<.001).

CONCLUSIONS

The NLP algorithm in our DeVTEcare identified VTE well across different clinical settings, especially in patients in surgery units, departments with low-risk VTE, and patients aged ≤65 years. This algorithm can help to inform accurate in-hospital VTE rates and enhance risk-classified VTE integrated care in future research.

Classification of neurologic outcomes from medical notes using natural language processing

Neurologic disability level at hospital discharge is an important outcome in many clinical research studies. Outside of clinical trials, neurologic outcomes must typically be extracted by labor intensive manual review of clinical notes in the electronic health record (EHR). To overcome this challenge, we set out to develop a natural language processing (NLP) approach that automatically reads clinical notes to determine neurologic outcomes, to make it possible to conduct larger scale neurologic outcomes studies. We obtained 7314 notes from 3632 patients hospitalized at two large Boston hospitals between January 2012 and June 2020, including discharge summaries (3485), occupational therapy (1472) and physical therapy (2357) notes. Fourteen clinical experts reviewed notes to assign scores on the Glasgow Outcome Scale (GOS) with 4 classes, namely 'good recovery', 'moderate disability', 'severe disability', and 'death' and on the Modified Rankin Scale (mRS), with 7 classes, namely 'no symptoms', 'no significant disability', 'slight disability', 'moderate disability', 'moderately severe disability', 'severe disability', and 'death'. For 428 patients' notes, 2 experts scored the cases generating interrater reliability estimates for GOS and mRS. After preprocessing and extracting features from the notes, we trained a multiclass logistic regression model using LASSO regularization and 5-fold cross validation for hyperparameter tuning. The model performed well on the test set, achieving a micro average area under the receiver operating characteristic and F-score of 0.94 (95% CI 0.93-0.95) and 0.77 (0.75-0.80) for GOS, and 0.90 (0.89-0.91) and 0.59 (0.57-0.62) for mRS, respectively. Our work demonstrates that an NLP algorithm can accurately assign neurologic outcomes based on free text clinical notes. This algorithm increases the scale of research on neurological outcomes that is possible with EHR data.

Computer-Based Natural Language Search Applied to the Electronic Medical Record for Tonsil Triage

OBJECTIVES

To determine significant histologic findings in tonsils and categorize clinical settings in which they occur to identify cases benefiting from histopathologic examination using a computer-based natural language search (NLS) applied to the electronic medical record.

METHODS

The pathology database was queried for tonsillectomy cases accessioned between 2002 and 2018. Tonsils with microscopic examination were reviewed, and indication for examination and diagnoses were tallied. Clinical risk of malignancy was correlated with findings. A NLS was used to interrogate preoperative clinical records of the same group of patients. The search identified cases at risk of significant histologic findings and was implemented as part of standard practice.

RESULTS

Of the 18,733 bilateral tonsillectomies identified in the pathology database, 494 were palatine tonsils that underwent microscopic examination, 134 had indications concerning for malignancy, and 14 had significant findings on histologic examination. When the NLS was applied to the medical record of the same group, 223 cases were identified as having risk of malignancy, including all flagged by surgeons and pathologists and 89 additional cases. Clinical implementation resulted in identification of all cases benefiting from examination.

CONCLUSIONS

A NLS applied to the electronic medical record to select tonsils for examination was superior to relying on surgeons and pathologists.

Diagnostic accuracy of clinical outcome prediction using nursing data in intensive care patients: A systematic review

BACKGROUND

Nursing data consist of observations of patients' conditions and information on nurses' clinical judgment based on critically ill patients' behavior and physiological signs. Nursing data in electronic health records were recently emphasized as important predictors of patients' deterioration but have not been systematically reviewed.

OBJECTIVE

We conducted a systematic review of prediction models using nursing data for clinical outcomes, such as prolonged hospital stay, readmission, and mortality in intensive care patients, compared to physiological data only. In addition, the type of nursing data used in prediction model developments was investigated.

DESIGN

A systematic review.

METHODS

PubMed, CINAHL, Cochrane CENTRAL, EMBASE, IEEE Xplore Digital Library, Web of Science, and Scopus were searched. Clinical outcome prediction models using nursing data for intensive care patients were included. Clinical outcomes were prolonged hospital stay, readmission, and mortality. Data were extracted from selected studies such as study design, data source, outcome definition, sample size, predictors, reference test, model development, model performance, and evaluation. The risk of bias and applicability was assessed using the Prediction model Risk of Bias Assessment Tool checklist. Descriptive summaries were produced based on paired forest plots and summary receiver operating characteristic curves.

RESULTS

Sixteen studies were included in the systematic review. The data types of predictors used in prediction models were categorized as physiological data, nursing data, and clinical notes. The types of nursing data consisted of nursing notes, assessments, documentation frequency, and flowsheet comments. The studies using physiological data as a reference test showed higher predictive performance in combined data or nursing data than in physiological data. The overall risk of bias indicated that most of the included studies have a high risk.

CONCLUSIONS

This study was conducted to identify and review the diagnostic accuracy of clinical outcome prediction using nursing data in intensive care patients. Most of the included studies developed models using nursing notes, and other studies used nursing assessments, documentation frequency, and flowsheet comments. Although the findings need careful interpretation due to the high risk of bias, the area under the curve scores of nursing data and combined data were higher than physiological data alone. It is necessary to establish a strategy in prediction modeling to utilize nursing data, clinical notes, and physiological data as predictors, considering the clinical context rather than physiological data alone.

REGISTRATION

The protocol for this study is registered with PROSPERO (registration number: CRD42021273319).

Development and Validation of a Machine Learning Approach Leveraging Real-World Clinical Narratives as a Predictor of Survival in Advanced Cancer

PURPOSE

Predicting short-term mortality in patients with advanced cancer remains challenging. Whether digitalized clinical text can be used to build models to enhance survival prediction in this population is unclear.

MATERIALS AND METHODS

We conducted a single-centered retrospective cohort study in patients with advanced solid tumors. Clinical correspondence authored by oncologists at the first patient encounter was extracted from the electronic medical records. Machine learning (ML) models were trained using narratives from the derivation cohort, before being tested on a temporal validation cohort at the same site. Performance was benchmarked against Eastern Cooperative Oncology Group performance status (PS), comparing ML models alone (comparison 1) or in combination with PS (comparison 2), assessed by areas under receiver operating characteristic curves (AUCs) for predicting vital status at 11 time points from 2 to 52 weeks.

RESULTS

ML models were built on the derivation cohort (4,791 patients from 2001 to April 2017) and tested on the validation cohort of 726 patients (May 2017-June 2019). In 441 patients (61%) where clinical narratives were available and PS was documented, ML models outperformed the predictivity of PS (mean AUC improvement, 0.039, P < .001, comparison 1). Inclusion of both clinical text and PS in ML models resulted in further improvement in prediction accuracy over PS with a mean AUC improvement of 0.050 ( P < .001, comparison 2); the AUC was > 0.80 at all assessed time points for models incorporating clinical text. Exploratory analysis of oncologist's narratives revealed recurring descriptors correlating with survival, including referral patterns, mobility, physical functions, and concomitant medications.

CONCLUSION

Applying ML to oncologists' narratives with or without including patient's PS significantly improved survival prediction to 12 months, suggesting the utility of clinical text in building prognostic support tools.

Factors associated with poor self-management documented in home health care narrative notes for patients with heart failure

BACKGROUND

Patients with heart failure (HF) who actively engage in their own self-management have better outcomes. Extracting data through natural language processing (NLP) holds great promise for identifying patients with or at risk of poor self-management.

OBJECTIVE

To identify home health care (HHC) patients with HF who have poor self-management using NLP of narrative notes, and to examine patient factors associated with poor self-management.

METHODS

An NLP algorithm was applied to extract poor self-management documentation using 353,718 HHC narrative notes of 9,710 patients with HF. Sociodemographic and structured clinical data were incorporated into multivariate logistic regression models to identify factors associated with poor self-management.

RESULTS

There were 758 (7.8%) patients in this sample identified as having notes with language describing poor HF self-management. Younger age (OR 0.982, 95% CI 0.976-0.987, p < .001), longer length of stay in HHC (OR 1.036, 95% CI 1.029- 1.043, p < .001), diagnosis of diabetes (OR 1.47, 95% CI 1.3-1.67, p < .001) and depression (OR 1.36, 95% CI 1.09-1.68, p < .01), impaired decision-making (OR 1.64, 95% CI 1.37-1.95, p < .001), smoking (OR 1.7, 95% CI 1.4-2.04, p < .001), and shortness of breath with exertion (OR 1.25, 95% CI 1.1-1.42, p < .01) were associated with poor self-management.

CONCLUSIONS

Patients with HF who have poor self-management can be identified from the narrative notes in HHC using novel NLP methods. Meaningful information about the self-management of patients with HF can support HHC clinicians in developing individualized care plans to improve self-management and clinical outcomes.

A multicenter prospective study on postoperative pulmonary complications prediction in geriatric patients with deep neural network model

Aim

Postoperative pulmonary complications (PPCs) can increase the risk of postoperative mortality, and the geriatric population has high incidence of PPCs. Early identification of high-risk geriatric patients is of great value for clinical decision making and prognosis improvement. Existing prediction models are based purely on structured data, and they lack predictive accuracy in geriatric patients. We aimed to develop and validate a deep neural network model based on combined natural language data and structured data for improving the prediction of PPCs in geriatric patients.

Methods

We consecutively enrolled patients aged ≥65 years who underwent surgery under general anesthesia at seven hospitals in China. Data from the West China Hospital of Sichuan University were used as the derivation dataset, and a deep neural network model was developed based on combined natural language data and structured data. Data from the six other hospitals were combined for external validation.

Results

The derivation dataset included 12,240 geriatric patients, and 1949(15.9%) patients developed PPCs. Our deep neural network model outperformed other machine learning models with an area under the precision-recall curve (AUPRC) of 0.657(95% confidence interval [CI], 0.655–0.658) and an area under the receiver operating characteristic curve (AUROC) of 0.884(95% CI, 0.883–0.885). The external dataset included 7579 patients, and 776(10.2%) patients developed PPCs. In external validation, the AUPRC was 0.632(95%CI, 0.632–0.633) and the AUROC was 0.889(95%CI, 0.888–0.889).

Conclusions

This study indicated that the deep neural network model based on combined natural language data and structured data could improve the prediction of PPCs in geriatric patients.

Risk Management In Intensive Care Units With Artificial Intelligence Technologies: Systematic Review of Prediction Models Using Electronic Health Records

Background and aim: Clinical risk assessments should be made to protect patients from negative outcomes, and the definition, frequency and severity of the risk should be determined. The information contained in the electronic health records (EHRs) can use in different areas such as risk prediction, estimation of treatment effect ect. Many prediction models using artificial intelligence (AI) technologies that can be used in risk assessment have been developed. The aim of this study is to bring together the researches on prediction models developed with AI technologies using the EHRs of patients hospitalized in the intensive care unit (ICU) and to evaluate them in terms of risk management in healthcare. Methods: The study restricted the search to the Web of Science, Pubmed, Science Direct, and Medline databases to retrieve research articles published in English in 2010 and after. Studies with a prediction model using data obtained from EHRs in the ICU are included. The study focused solely on research conducted in ICU to predict a health condition that poses a significant risk to patient safety using artificial intellegence (AI) technologies. Results: Recognized prediction subcategories were mortality (n=6), sepsis (n=4), pressure ulcer (n=4), acute kidney injury (n=3), and other areas (n=10). It has been found that EHR-based prediction models are good risk management and decision support tools and adoption of such models in ICUs may reduce the prevalence of adverse conditions. Conclusions: The article results remarks that developed models was found to have higher performance and better selectivity than previously developed risk models, so they are better at predicting risks and serious adverse events in ICU. It is recommended to use AI based prediction models developed using EHRs in risk management studies. Future work is still needed to researches to predict different health conditions risks.

Feasibility of Extracting Meaningful Patient Centered Outcomes From the Electronic Health Record Following Critical Illness in the Elderly

Background

Meaningful patient centered outcomes of critical illness such as functional status, cognition and mental health are studied using validated measurement tools that may often be impractical outside the research setting. The Electronic health record (EHR) contains a plethora of information pertaining to these domains. We sought to determine how feasible and reliable it is to assess meaningful patient centered outcomes from the EHR.

Methods

Two independent investigators reviewed EHR of a random sample of ICU patients looking at documented assessments of trajectory of functional status, cognition, and mental health. Cohen's kappa was used to measure agreement between 2 reviewers. Post ICU health in these domains 12 month after admission was compared to pre- ICU health in the 12 months prior to assess qualitatively whether a patient's condition was “better,” “unchanged” or “worse.” Days alive and out of hospital/health care facility was a secondary outcome.

Results

Thirty six of the 41 randomly selected patients (88%) survived critical illness. EHR contained sufficient information to determine the difference in health status before and after critical illness in most survivors (86%). Decline in functional status (36%), cognition (11%), and mental health (11%) following ICU admission was observed compared to premorbid baseline. Agreement between reviewers was excellent (kappa ranging from 0.966 to 1). Eighteen patients (44%) remained home after discharge from hospital and rehabilitation during the 12- month follow up.

Conclusion

We demonstrated the feasibility and reliability of assessing the trajectory of changes in functional status, cognition, and selected mental health outcomes from EHR of critically ill patients. If validated in a larger, representative sample, these outcomes could be used alongside survival in quality improvement studies and pragmatic clinical trials.

Artificial intelligence and clinical deterioration

PURPOSE OF REVIEW

To provide an overview of the systems being used to identify and predict clinical deterioration in hospitalised patients, with focus on the current and future role of artificial intelligence (AI).

RECENT FINDINGS

There are five leading AI driven systems in this field: the Advanced Alert Monitor (AAM), the electronic Cardiac Arrest Risk Triage (eCART) score, Hospital wide Alert Via Electronic Noticeboard, the Mayo Clinic Early Warning Score, and the Rothman Index (RI). Each uses Electronic Patient Record (EPR) data and machine learning to predict adverse events. Less mature but relevant evolutions are occurring in the fields of Natural Language Processing, Time and Motion Studies, AI Sepsis and COVID-19 algorithms.

SUMMARY

Research-based AI-driven systems to predict clinical deterioration are increasingly being developed, but few are being implemented into clinical workflows. Escobar et al. (AAM) provide the current gold standard for robust model development and implementation methodology. Multiple technologies show promise, however, the pathway to meaningfully affect patient outcomes remains challenging.

Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

Background

Machine learning (ML) achieves better predictions of postoperative mortality than previous prediction tools. Free-text descriptions of the preoperative diagnosis and the planned procedure are available preoperatively. Because reading these descriptions helps anesthesiologists evaluate the risk of the surgery, we hypothesized that deep learning (DL) models with unstructured text could improve postoperative mortality prediction. However, it is challenging to extract meaningful concept embeddings from this unstructured clinical text.

Objective

This study aims to develop a fusion DL model containing structured and unstructured features to predict the in-hospital 30-day postoperative mortality before surgery. ML models for predicting postoperative mortality using preoperative data with or without free clinical text were assessed.

Methods

We retrospectively collected preoperative anesthesia assessments, surgical information, and discharge summaries of patients undergoing general and neuraxial anesthesia from electronic health records (EHRs) from 2016 to 2020. We first compared the deep neural network (DNN) with other models using the same input features to demonstrate effectiveness. Then, we combined the DNN model with bidirectional encoder representations from transformers (BERT) to extract information from clinical texts. The effects of adding text information on the model performance were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Statistical significance was evaluated using P<.05.

Results

The final cohort contained 121,313 patients who underwent surgeries. A total of 1562 (1.29%) patients died within 30 days of surgery. Our BERT-DNN model achieved the highest AUROC (0.964, 95% CI 0.961-0.967) and AUPRC (0.336, 95% CI 0.276-0.402). The AUROC of the BERT-DNN was significantly higher compared to logistic regression (AUROC=0.952, 95% CI 0.949-0.955) and the American Society of Anesthesiologist Physical Status (ASAPS AUROC=0.892, 95% CI 0.887-0.896) but not significantly higher compared to the DNN (AUROC=0.959, 95% CI 0.956-0.962) and the random forest (AUROC=0.961, 95% CI 0.958-0.964). The AUPRC of the BERT-DNN was significantly higher compared to the DNN (AUPRC=0.319, 95% CI 0.260-0.384), the random forest (AUPRC=0.296, 95% CI 0.239-0.360), logistic regression (AUPRC=0.276, 95% CI 0.220-0.339), and the ASAPS (AUPRC=0.149, 95% CI 0.107-0.203).

Conclusions

Our BERT-DNN model has an AUPRC significantly higher compared to previously proposed models using no text and an AUROC significantly higher compared to logistic regression and the ASAPS. This technique helps identify patients with higher risk from the surgical description text in EHRs.

Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study

PURPOSE

Oncologists may overestimate prognosis for patients with cancer, leading to delayed or missed conversations about patients' goals and subsequent low-quality end-of-life care. Machine learning algorithms may accurately predict mortality risk in cancer, but it is unclear how oncology clinicians would use such algorithms in practice.

METHODS

The purpose of this qualitative study was to assess oncology clinicians' perceptions on the utility and barriers of machine learning prognostic algorithms to prompt advance care planning. Participants included medical oncology physicians and advanced practice providers (APPs) practicing in tertiary and community practices within a large academic healthcare system. Transcripts were coded and analyzed inductively using NVivo software.

RESULTS

The study included 29 oncology clinicians (19 physicians, 10 APPs) across 6 practice sites (1 tertiary, 5 community) in the USA. Fourteen participants had previously had exposure to an automated machine learning-based prognostic algorithm as part of a pragmatic randomized trial. Clinicians believed that there was utility for algorithms in validating their own intuition about prognosis and prompting conversations about patient goals and preferences. However, this enthusiasm was tempered by concerns about algorithm accuracy, over-reliance on algorithm predictions, and the ethical implications around disclosure of an algorithm prediction. There was significant variation in tolerance for false positive vs. false negative predictions.

CONCLUSION

While oncologists believe there are applications for advanced prognostic algorithms in routine care of patients with cancer, they are concerned about algorithm accuracy, confirmation and automation biases, and ethical issues of prognostic disclosure.

Psychosis Relapse Prediction Leveraging Electronic Health Records Data and Natural Language Processing Enrichment Methods

BACKGROUND

Identifying patients at a high risk of psychosis relapse is crucial for early interventions. A relevant psychiatric clinical context is often recorded in clinical notes; however, the utilization of unstructured data remains limited. This study aimed to develop psychosis-relapse prediction models using various types of clinical notes and structured data.

METHODS

Clinical data were extracted from the electronic health records of the Ajou University Medical Center in South Korea. The study population included patients with psychotic disorders, and outcome was psychosis relapse within 1 year. Using only structured data, we developed an initial prediction model, then three natural language processing (NLP)-enriched models using three types of clinical notes (psychological tests, admission notes, and initial nursing assessment) and one complete model. Latent Dirichlet Allocation was used to cluster the clinical context into similar topics. All models applied the least absolute shrinkage and selection operator logistic regression algorithm. We also performed an external validation using another hospital database.

RESULTS

A total of 330 patients were included, and 62 (18.8%) experienced psychosis relapse. Six predictors were used in the initial model and 10 additional topics from Latent Dirichlet Allocation processing were added in the enriched models. The model derived from all notes showed the highest value of the area under the receiver operating characteristic (AUROC = 0.946) in the internal validation, followed by models based on the psychological test notes, admission notes, initial nursing assessments, and structured data only (0.902, 0.855, 0.798, and 0.784, respectively). The external validation was performed using only the initial nursing assessment note, and the AUROC was 0.616.

CONCLUSIONS

We developed prediction models for psychosis relapse using the NLP-enrichment method. Models using clinical notes were more effective than models using only structured data, suggesting the importance of unstructured data in psychosis prediction.

OUP accepted manuscript

Objective

This systematic review aims to assess how information from unstructured text is used to develop and validate clinical prognostic prediction models. We summarize the prediction problems and methodological landscape and determine whether using text data in addition to more commonly used structured data improves the prediction performance.

Materials and Methods

We searched Embase, MEDLINE, Web of Science, and Google Scholar to identify studies that developed prognostic prediction models using information extracted from unstructured text in a data-driven manner, published in the period from January 2005 to March 2021. Data items were extracted, analyzed, and a meta-analysis of the model performance was carried out to assess the added value of text to structured-data models.

Results

We identified 126 studies that described 145 clinical prediction problems. Combining text and structured data improved model performance, compared with using only text or only structured data. In these studies, a wide variety of dense and sparse numeric text representations were combined with both deep learning and more traditional machine learning methods. External validation, public availability, and attention for the explainability of the developed models were limited.

Conclusion

The use of unstructured text in the development of prognostic prediction models has been found beneficial in addition to structured data in most studies. The text data are source of valuable information for prediction model development and should not be neglected. We suggest a future focus on explainability and external validation of the developed models, promoting robust and trustworthy prediction models in clinical practice.

Automated model versus treating physician for predicting survival time of patients with metastatic cancer

OBJECTIVE

Being able to predict a patient's life expectancy can help doctors and patients prioritize treatments and supportive care. For predicting life expectancy, physicians have been shown to outperform traditional models that use only a few predictor variables. It is possible that a machine learning model that uses many predictor variables and diverse data sources from the electronic medical record can improve on physicians' performance. For patients with metastatic cancer, we compared accuracy of life expectancy predictions by the treating physician, a machine learning model, and a traditional model.

MATERIALS AND METHODS

A machine learning model was trained using 14 600 metastatic cancer patients' data to predict each patient's distribution of survival time. Data sources included note text, laboratory values, and vital signs. From 2015-2016, 899 patients receiving radiotherapy for metastatic cancer were enrolled in a study in which their radiation oncologist estimated life expectancy. Survival predictions were also made by the machine learning model and a traditional model using only performance status. Performance was assessed with area under the curve for 1-year survival and calibration plots.

RESULTS

The radiotherapy study included 1190 treatment courses in 899 patients. A total of 879 treatment courses in 685 patients were included in this analysis. Median overall survival was 11.7 months. Physicians, machine learning model, and traditional model had area under the curve for 1-year survival of 0.72 (95% CI 0.63-0.81), 0.77 (0.73-0.81), and 0.68 (0.65-0.71), respectively.

CONCLUSIONS

The machine learning model's predictions were more accurate than those of the treating physician or a traditional model.

Modern Clinical Text Mining: A Guide and Review

Electronic health records (EHRs) are becoming a vital source of data for healthcare quality improvement, research, and operations. However, much of the most valuable information contained in EHRs remains buried in unstructured text. The field of clinical text mining has advanced rapidly in recent years, transitioning from rule-based approaches to machine learning and, more recently, deep learning. With new methods come new challenges, however, especially for those new to the field. This review provides an overview of clinical text mining for those who are encountering it for the first time (e.g., physician researchers, operational analytics teams, machine learning scientists from other domains). While not a comprehensive survey, this review describes the state of the art, with a particular focus on new tasks and methods developed over the past few years. It also identifies key barriers between these remarkable technical advances and the practical realities of implementation in health systems and in industry.

Deep learning to predict long-term mortality in patients requiring 7 days of mechanical ventilation

Background

Among patients with acute respiratory failure requiring prolonged mechanical ventilation, tracheostomies are typically placed after approximately 7 to 10 days. Yet half of patients admitted to the intensive care unit receiving tracheostomy will die within a year, often within three months. Existing mortality prediction models for prolonged mechanical ventilation, such as the ProVent Score, have poor sensitivity and are not applied until after 14 days of mechanical ventilation. We developed a model to predict 3-month mortality in patients requiring more than 7 days of mechanical ventilation using deep learning techniques and compared this to existing mortality models.

Methods

Retrospective cohort study. Setting: The Medical Information Mart for Intensive Care III Database. Patients: All adults requiring ≥ 7 days of mechanical ventilation. Measurements: A neural network model for 3-month mortality was created using process-of-care variables, including demographic, physiologic and clinical data. The area under the receiver operator curve (AUROC) was compared to the ProVent model at predicting 3 and 12-month mortality. Shapley values were used to identify the variables with the greatest contributions to the model.

Results

There were 4,334 encounters divided into a development cohort (n = 3467) and a testing cohort (n = 867). The final deep learning model included 250 variables and had an AUROC of 0.74 for predicting 3-month mortality at day 7 of mechanical ventilation versus 0.59 for the ProVent model. Older age and elevated Simplified Acute Physiology Score II (SAPS II) Score on intensive care unit admission had the largest contribution to predicting mortality.

Discussion

We developed a deep learning prediction model for 3-month mortality among patients requiring ≥ 7 days of mechanical ventilation using a neural network approach utilizing readily available clinical variables. The model outperforms the ProVent model for predicting mortality among patients requiring ≥ 7 days of mechanical ventilation. This model requires external validation.

Impact of Different Approaches to Preparing Notes for Analysis With Natural Language Processing on the Performance of Prediction Models in Intensive Care

Supplemental Digital Content is available in the text.

OBJECTIVES:

To evaluate whether different approaches in note text preparation (known as preprocessing) can impact machine learning model performance in the case of mortality prediction ICU.

DESIGN:

Clinical note text was used to build machine learning models for adults admitted to the ICU. Preprocessing strategies studied were none (raw text), cleaning text, stemming, term frequency-inverse document frequency vectorization, and creation of n-grams. Model performance was assessed by the area under the receiver operating characteristic curve. Models were trained and internally validated on University of California San Francisco data using 10-fold cross validation. These models were then externally validated on Beth Israel Deaconess Medical Center data.

SETTING:

ICUs at University of California San Francisco and Beth Israel Deaconess Medical Center.

SUBJECTS:

Ten thousand patients in the University of California San Francisco training and internal testing dataset and 27,058 patients in the external validation dataset, Beth Israel Deaconess Medical Center.

INTERVENTIONS:

None.

MEASUREMENTS AND MAIN RESULTS:

Mortality rate at Beth Israel Deaconess Medical Center and University of California San Francisco was 10.9% and 7.4%, respectively. Data are presented as area under the receiver operating characteristic curve (95% CI) for models validated at University of California San Francisco and area under the receiver operating characteristic curve for models validated at Beth Israel Deaconess Medical Center. Models built and trained on University of California San Francisco data for the prediction of inhospital mortality improved from the raw note text model (AUROC, 0.84; CI, 0.80–0.89) to the term frequency-inverse document frequency model (AUROC, 0.89; CI, 0.85–0.94). When applying the models developed at University of California San Francisco to Beth Israel Deaconess Medical Center data, there was a similar increase in model performance from raw note text (area under the receiver operating characteristic curve at Beth Israel Deaconess Medical Center: 0.72) to the term frequency-inverse document frequency model (area under the receiver operating characteristic curve at Beth Israel Deaconess Medical Center: 0.83).

CONCLUSIONS:

Differences in preprocessing strategies for note text impacted model discrimination. Completing a preprocessing pathway including cleaning, stemming, and term frequency-inverse document frequency vectorization resulted in the preprocessing strategy with the greatest improvement in model performance. Further study is needed, with particular emphasis on how to manage author implicit bias present in note text, before natural language processing algorithms are implemented in the clinical setting.

Predictors of death and new disability after critical illness: a multicentre prospective cohort study

PURPOSE

This study aimed to determine the prevalence and predictors of death or new disability following critical illness.

METHODS

Prospective, multicentre cohort study conducted in six metropolitan intensive care units (ICU). Participants were adults admitted to the ICU who received more than 24 h of mechanical ventilation. The primary outcome was death or new disability at 6 months, with new disability defined by a 10% increase in the WHODAS 2.0.

RESULTS

Of 628 patients with the primary outcome available (median age of 62 [49-71] years, 379 [61.0%] had a medical admission and 370 (58.9%) died or developed new disability by 6 months. Independent predictors of death or new disability included age [OR 1.02 (1.01-1.03), P = 0.001], higher severity of illness (APACHE III) [OR 1.02 (1.01-1.03), P < 0.001] and admission diagnosis. Compared to patients with a surgical admission diagnosis, patients with a cardiac arrest [OR (95% CI) 4.06 (1.89-8.68), P < 0.001], sepsis [OR (95% CI) 2.43 (1.32-4.47), P = 0.004], or trauma [OR (95% CI) 6.24 (3.07-12.71), P < 0.001] diagnosis had higher odds of death or new disability, while patients with a lung transplant [OR (95% CI) 0.21 (0.07-0.58), P = 0.003] diagnosis had lower odds. A model including these three variables had good calibration (Brier score 0.20) and acceptable discriminative power with an area under the receiver operating characteristic curve of 0.76 (95% CI 0.72-0.80).

CONCLUSION

Less than half of all patients mechanically ventilated for more than 24 h were alive and free of new disability at 6 months after admission to ICU. A model including age, illness severity and admission diagnosis has acceptable discriminative ability to predict death or new disability at 6 months.

Application of Artificial Intelligence for the Diagnosis and Treatment of Liver Diseases

Modern medical care produces large volumes of multimodal patient data, which many clinicians struggle to process and synthesize into actionable knowledge. In recent years, artificial intelligence (AI) has emerged as an effective tool in this regard. The field of hepatology is no exception, with a growing number of studies published that apply AI techniques to the diagnosis and treatment of liver diseases. These have included machine-learning algorithms (such as regression models, Bayesian networks, and support vector machines) to predict disease progression, the presence of complications, and mortality; deep-learning algorithms to enable rapid, automated interpretation of radiologic and pathologic images; and natural-language processing to extract clinically meaningful concepts from vast quantities of unstructured data in electronic health records. This review article will provide a comprehensive overview of hepatology-focused AI research, discuss some of the barriers to clinical implementation and adoption, and suggest future directions for the field.

Large-scale identification of aortic stenosis and its severity using natural language processing on electronic health records

Background

Objective

Methods

Results

Conclusion

Quantification of BERT Diagnosis Generalizability Across Medical Specialties Using Semantic Dataset Distance

Deep learning models in healthcare may fail to generalize on data from unseen corpora. Additionally, no quantitative metric exists to tell how existing models will perform on new data. Previous studies demonstrated that NLP models of medical notes generalize variably between institutions, but ignored other levels of healthcare organization. We measured SciBERT diagnosis sentiment classifier generalizability between medical specialties using EHR sentences from MIMIC-III. Models trained on one specialty performed better on internal test sets than mixed or external test sets (mean AUCs 0.92, 0.87, and 0.83, respectively; p = 0.016). When models are trained on more specialties, they have better test performances (p < 1e-4). Model performance on new corpora is directly correlated to the similarity between train and test sentence content (p < 1e-4). Future studies should assess additional axes of generalization to ensure deep learning models fulfil their intended purpose across institutions, specialties, and practices.

Mortality prediction for patients with acute respiratory distress syndrome based on machine learning: a population-based study

Background

Traditional scoring systems for patients' outcome prediction in intensive care units such as Oxygenation Saturation Index (OSI) and Oxygenation Index (OI) may not reliably predict the clinical prognosis of patients with acute respiratory distress syndrome (ARDS). Thus, none of them have been widely accepted for mortality prediction in ARDS. This study aimed to develop and validate a mortality prediction method for patients with ARDS based on machine learning using the Medical Information Mart for Intensive Care (MIMIC-III) and Telehealth Intensive Care Unit (eICU) Collaborative Research Database (eICU-CRD) databases.

Methods

Patients with ARDS were selected based on the Berlin definition in MIMIC-III and eICU-CRD databases. The APPS score (using age, PaO₂/FiO₂, and plateau pressure), Simplified Acute Physiology Score II (SAPS-II), Sepsis-related Organ Failure Assessment (SOFA), OSI, and OI were calculated. With MIMIC-III data, a mortality prediction model was built based on the random forest (RF) algorithm, and the performance was compared to those of existing scoring systems based on logistic regression. The performance of the proposed RF method was also validated with the combined MIMIC-III and eICU-CRD data. The performance of mortality prediction was evaluated by using the area under the receiver operating characteristics curve (AUROC) and performing calibration using the Hosmer-Lemeshow test.

Results

With the MIMIC-III dataset (308 patients, for comparisons with the existing scoring systems), the RF model predicted the in-hospital mortality, 30-day mortality, and 1-year mortality with an AUROC of 0.891, 0.883, and 0.892, respectively, which were significantly higher than those of the SAPS-II, APPS, OSI, and OI (all P<0.001). In the multi-source validation (the combined dataset of 2,235 patients in MIMIC-III and 331 patients in eICU-CRD), the RF model achieved an AUROC of 0.905 and 0.736 for predicting in-hospital mortality for the MIMIC-III and eICU-CRD datasets, respectively. The calibration plots suggested good fits for our RF model and these scoring systems for predicting mortality. The platelet count and lactate level were the strongest predictive variables for predicting in-hospital mortality.

Conclusions

Compared to the existing scoring systems, machine learning significantly improved performance for predicting ARDS mortality. Validation with multi-source datasets showed a relatively robust generalisation ability of our prediction model.

Performance of intensive care unit severity scoring systems across different ethnicities in the USA: a retrospective observational study

BACKGROUND

Despite wide use of severity scoring systems for case-mix determination and benchmarking in the intensive care unit (ICU), the possibility of scoring bias across ethnicities has not been examined. Guidelines on the use of illness severity scores to inform triage decisions for allocation of scarce resources, such as mechanical ventilation, during the current COVID-19 pandemic warrant examination for possible bias in these models. We investigated the performance of the severity scoring systems Acute Physiology and Chronic Health Evaluation IVa (APACHE IVa), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA) across four ethnicities in two large ICU databases to identify possible ethnicity-based bias.

METHODS

Data from the electronic ICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care III (MIMIC-III) database, built from patient episodes in the USA from 2014-15 and 2001-12, respectively, were analysed for score performance in Asian, Black, Hispanic, and White people after appropriate exclusions. Hospital mortality was the outcome of interest. Discrimination and calibration were determined for all three scoring systems in all four groups, using area under receiver operating characteristic (AUROC) curve for different ethnicities to assess discrimination, and standardised mortality ratio (SMR) or proxy measures to assess calibration.

FINDINGS

We analysed 166 751 participants (122 919 eICU-CRD and 43 832 MIMIC-III). Although measurements of discrimination were significantly different among the groups (AUROC ranging from 0·86 to 0·89 [p=0·016] with APACHE IVa and from 0·75 to 0·77 [p=0·85] with OASIS), they did not display any discernible systematic patterns of bias. However, measurements of calibration indicated persistent, and in some cases statistically significant, patterns of difference between Hispanic people (SMR 0·73 with APACHE IVa and 0·64 with OASIS) and Black people (0·67 and 0·68) versus Asian people (0·77 and 0·95) and White people (0·76 and 0·81). Although calibrations were imperfect for all groups, the scores consistently showed a pattern of overpredicting mortality for Black people and Hispanic people. Similar results were seen using SOFA scores across the two databases.

INTERPRETATION

The systematic differences in calibration across ethnicities suggest that illness severity scores reflect statistical bias in their predictions of mortality.

FUNDING

There was no specific funding for this study.

Classification of the Disposition of Patients Hospitalized with COVID-19: Reading Discharge Summaries Using Natural Language Processing

Background

Medical notes are a rich source of patient data; however, the nature of unstructured text has largely precluded the use of these data for large retrospective analyses. Transforming clinical text into structured data can enable large-scale research studies with electronic health records (EHR) data. Natural language processing (NLP) can be used for text information retrieval, reducing the need for labor-intensive chart review. Here we present an application of NLP to large-scale analysis of medical records at 2 large hospitals for patients hospitalized with COVID-19.

Objective

Our study goal was to develop an NLP pipeline to classify the discharge disposition (home, inpatient rehabilitation, skilled nursing inpatient facility [SNIF], and death) of patients hospitalized with COVID-19 based on hospital discharge summary notes.

Methods

Text mining and feature engineering were applied to unstructured text from hospital discharge summaries. The study included patients with COVID-19 discharged from 2 hospitals in the Boston, Massachusetts area (Massachusetts General Hospital and Brigham and Women’s Hospital) between March 10, 2020, and June 30, 2020. The data were divided into a training set (70%) and hold-out test set (30%). Discharge summaries were represented as bags-of-words consisting of single words (unigrams), bigrams, and trigrams. The number of features was reduced during training by excluding n-grams that occurred in fewer than 10% of discharge summaries, and further reduced using least absolute shrinkage and selection operator (LASSO) regularization while training a multiclass logistic regression model. Model performance was evaluated using the hold-out test set.

Results

The study cohort included 1737 adult patients (median age 61 [SD 18] years; 55% men; 45% White and 16% Black; 14% nonsurvivors and 61% discharged home). The model selected 179 from a vocabulary of 1056 engineered features, consisting of combinations of unigrams, bigrams, and trigrams. The top features contributing most to the classification by the model (for each outcome) were the following: “appointments specialty,” “home health,” and “home care” (home); “intubate” and “ARDS” (inpatient rehabilitation); “service” (SNIF); “brief assessment” and “covid” (death). The model achieved a micro-average area under the receiver operating characteristic curve value of 0.98 (95% CI 0.97-0.98) and average precision of 0.81 (95% CI 0.75-0.84) in the testing set for prediction of discharge disposition.

Conclusions

A supervised learning–based NLP approach is able to classify the discharge disposition of patients hospitalized with COVID-19. This approach has the potential to accelerate and increase the scale of research on patients’ discharge disposition that is possible with EHR data.