Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses

In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensive human evaluation framework. Employing ChatGPT-3.5-turbo generative output, we correlated human judgments with each metric. None of the metrics demonstrated high alignment; however, the SapBERT score-a Unified Medical Language System (UMLS)- showed the best results. This underscores the importance of incorporating domain-specific knowledge into evaluation efforts. Our work reveals the deficiency in quality evaluations for generated text and introduces our comprehensive human evaluation framework as a baseline. Future efforts should prioritize integrating medical knowledge databases to enhance the alignment of automated metrics, particularly focusing on refining the SapBERT score for improved assessments.

Automated stratification of trauma injury severity across multiple body regions using multi-modal, multi-class machine learning models

OBJECTIVE

The timely stratification of trauma injury severity can enhance the quality of trauma care but it requires intense manual annotation from certified trauma coders. The objective of this study is to develop machine learning models for the stratification of trauma injury severity across various body regions using clinical text and structured electronic health records (EHRs) data.

MATERIALS AND METHODS

Our study utilized clinical documents and structured EHR variables linked with the trauma registry data to create 2 machine learning models with different approaches to representing text. The first one fuses concept unique identifiers (CUIs) extracted from free text with structured EHR variables, while the second one integrates free text with structured EHR variables. Temporal validation was undertaken to ensure the models' temporal generalizability. Additionally, analyses to assess the variable importance were conducted.

RESULTS

Both models demonstrated impressive performance in categorizing leg injuries, achieving high accuracy with macro-F1 scores of over 0.8. Additionally, they showed considerable accuracy, with macro-F1 scores exceeding or near 0.7, in assessing injuries in the areas of the chest and head. We showed in our variable importance analysis that the most important features in the model have strong face validity in determining clinically relevant trauma injuries.

DISCUSSION

The CUI-based model achieves comparable performance, if not higher, compared to the free-text-based model, with reduced complexity. Furthermore, integrating structured EHR data improves performance, particularly when the text modalities are insufficiently indicative.

CONCLUSIONS

Our multi-modal, multiclass models can provide accurate stratification of trauma injury severity and clinically relevant interpretations.

Performance of International Classification of Disease-10 codes in detecting emergency department patients with opioid misuse

BACKGROUND AND AIMS

Accurate case discovery is critical for disease surveillance, resource allocation and research. International Classification of Disease (ICD) diagnosis codes are commonly used for this purpose. We aimed to determine the sensitivity, specificity and positive predictive value (PPV) of ICD-10 codes for opioid misuse case discovery in the emergency department (ED) setting.

DESIGN AND SETTING

Retrospective cohort study of ED encounters from January 2018 to December 2020 at an urban academic hospital in the United States. A sample of ED encounters enriched for opioid misuse was developed by oversampling ED encounters with positive urine opiate screens or pre-existing opioid-related diagnosis codes in addition to other opioid misuse risk factors.

CASES

A total of 1200 randomly selected encounters were annotated by research staff for the presence of opioid misuse within health record documentation using a 5-point scale for likelihood of opioid misuse and dichotomized into cohorts of opioid misuse and no opioid misuse.

MEASUREMENTS

Using manual annotation as ground truth, the sensitivity and specificity of ICD-10 codes entered during the encounter were determined with PPV adjusted for oversampled data. Metrics were also determined by disposition subgroup: discharged home or admitted.

FINDINGS

There were 541 encounters annotated as opioid misuse and 617 with no opioid misuse. The majority were males (54.4%), average age was 47 years and 68.5% were discharged directly from the ED. The sensitivity of ICD-10 codes was 0.56 (95% confidence interval [CI], 0.51-0.60), specificity 0.99 (95% CI, 0.97-0.99) and adjusted PPV 0.78 (95% CI, 0.65-0.92). The sensitivity was higher for patients discharged from the ED (0.65; 95% CI, 0.60-0.69) than those admitted (0.31; 95% CI, 0.24-0.39).

CONCLUSIONS

International Classification of Disease-10 codes appear to have low sensitivity but high specificity and positive predictive value in detecting opioid misuse among emergency department patients in the United States.

LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction

Natural Language Processing (NLP) is a study of automated processing of text data. Application of NLP in the clinical domain is important due to the rich unstructured information implanted in clinical documents, which often remains inaccessible in structured data. Empowered by the recent advance of language models (LMs), there is a growing interest in their application within the clinical domain. When applying NLP methods to a certain domain, the role of benchmark datasets are crucial as benchmark datasets not only guide the selection of best-performing models but also enable assessing of the reliability of the generated outputs. Despite the recent availability of LMs capable of longer context, benchmark datasets targeting long clinical document classification tasks are absent. To address this issue, we propose LCD benchmark, a benchmark for the task of predicting 30-day out-of-hospital mortality using discharge notes of MIMIC-IV and statewide death data. Our notes have a median word count of 1687 and an interquartile range of 1308 to 2169. We evaluated this benchmark dataset using baseline models, from bag-of-words and CNN to Hierarchical Transformer and an open-source instruction-tuned large language model. Additionally, we provide a comprehensive analysis of the model outputs, including manual review and visualization of model weights, to offer insights into their predictive capabilities and limitations. We expect LCD benchmarks to become a resource for the development of advanced supervised models, prompting methods, or the foundation models themselves, tailored for clinical text. The benchmark dataset is available at https://github.com/Machine-Learning-for-Medical-Language/long-clinical-doc.

Multicenter Development and Prospective Validation of eCARTv5: A Gradient Boosted Machine Learning Early Warning Score

Rationale

Early detection of clinical deterioration using early warning scores may improve outcomes. However, most implemented scores were developed using logistic regression, only underwent retrospective internal validation, and were not tested in important patient subgroups.

Objectives

To develop a gradient boosted machine model (eCARTv5) for identifying clinical deterioration and then validate externally, test prospectively, and evaluate across patient subgroups.

Methods

All adult patients hospitalized on the wards in seven hospitals from 2008- 2022 were used to develop eCARTv5, with demographics, vital signs, clinician documentation, and laboratory values utilized to predict intensive care unit transfer or death in the next 24 hours. The model was externally validated retrospectively in 21 hospitals from 2009-2023 and prospectively in 10 hospitals from February to May 2023. eCARTv5 was compared to the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS) using the area under the receiver operating characteristic curve (AUROC).

Measurements and Main Results

The development cohort included 901,491 admissions, the retrospective validation cohort included 1,769,461 admissions, and the prospective validation cohort included 46,330 admissions. In retrospective validation, eCART had the highest AUROC (0.835; 95%CI 0.834, 0.835), followed by NEWS (0.766 (95%CI 0.766, 0.767)), and MEWS (0.704 (95%CI 0.703, 0.704)). eCART's performance remained high (AUROC ≥0.80) across a range of patient demographics, clinical conditions, and during prospective validation.

Conclusions

We developed eCARTv5, which accurately identifies early clinical deterioration in hospitalized ward patients. Our model performed better than the NEWS and MEWS retrospectively, prospectively, and across a range of subgroups.

Alcohol Withdrawal Severity Measures for Identifying Patients Requiring High-Intensity Care

OBJECTIVES

Alcohol withdrawal syndrome (AWS) may progress to require high-intensity care. Approaches to identify hospitalized patients with AWS who received higher level of care have not been previously examined. This study aimed to examine the utility of Clinical Institute Withdrawal Assessment Alcohol Revised (CIWA-Ar) for alcohol scale scores and medication doses for alcohol withdrawal management in identifying patients who received high-intensity care.

DESIGN

A multicenter observational cohort study of hospitalized adults with alcohol withdrawal.

SETTING

University of Chicago Medical Center and University of Wisconsin Hospital.

PATIENTS

Inpatient encounters between November 2008 and February 2022 with a CIWA-Ar score greater than 0 and benzodiazepine or barbiturate administered within the first 24 hours. The primary composite outcome was patients who progressed to high-intensity care (intermediate care or ICU).

INTERVENTIONS

None.

MAIN RESULTS

Among the 8742 patients included in the study, 37.5% (n = 3280) progressed to high-intensity care. The odds ratio for the composite outcome increased above 1.0 when the CIWA-Ar score was 24. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at this threshold were 0.12 (95% CI, 0.11-0.13), 0.95 (95% CI, 0.94-0.95), 0.58 (95% CI, 0.54-0.61), and 0.64 (95% CI, 0.63-0.65), respectively. The OR increased above 1.0 at a 24-hour lorazepam milligram equivalent dose cutoff of 15 mg. The sensitivity, specificity, PPV, and NPV at this threshold were 0.16 (95% CI, 0.14-0.17), 0.96 (95% CI, 0.95-0.96), 0.68 (95% CI, 0.65-0.72), and 0.65 (95% CI, 0.64-0.66), respectively.

CONCLUSIONS

Neither CIWA-Ar scores nor medication dose cutoff points were effective measures for identifying patients with alcohol withdrawal who received high-intensity care. Research studies for examining outcomes in patients who deteriorate with AWS will require better methods for cohort identification.

Validation of SeptiCyte RAPID to Discriminate Sepsis from Non-Infectious Systemic Inflammation

(1) Background: SeptiCyte RAPID is a molecular test for discriminating sepsis from non-infectious systemic inflammation, and for estimating sepsis probabilities. The objective of this study was the clinical validation of SeptiCyte RAPID, based on testing retrospectively banked and prospectively collected patient samples. (2) Methods: The cartridge-based SeptiCyte RAPID test accepts a PAXgene blood RNA sample and provides sample-to-answer processing in ~1 h. The test output (SeptiScore, range 0-15) falls into four interpretation bands, with higher scores indicating higher probabilities of sepsis. Retrospective (N = 356) and prospective (N = 63) samples were tested from adult patients in ICU who either had the systemic inflammatory response syndrome (SIRS), or were suspected of having/diagnosed with sepsis. Patients were clinically evaluated by a panel of three expert physicians blinded to the SeptiCyte test results. Results were interpreted under either the Sepsis-2 or Sepsis-3 framework. (3) Results: Under the Sepsis-2 framework, SeptiCyte RAPID performance for the combined retrospective and prospective cohorts had Areas Under the ROC Curve (AUCs) ranging from 0.82 to 0.85, a negative predictive value of 0.91 (sensitivity 0.94) for SeptiScore Band 1 (score range 0.1-5.0; lowest risk of sepsis), and a positive predictive value of 0.81 (specificity 0.90) for SeptiScore Band 4 (score range 7.4-15; highest risk of sepsis). Performance estimates for the prospective cohort ranged from AUC 0.86-0.95. For physician-adjudicated sepsis cases that were blood culture (+) or blood, urine culture (+)(+), 43/48 (90%) of SeptiCyte scores fell in Bands 3 or 4. In multivariable analysis with up to 14 additional clinical variables, SeptiScore was the most important variable for sepsis diagnosis. A comparable performance was obtained for the majority of patients reanalyzed under the Sepsis-3 definition, although a subgroup of 16 patients was identified that was called septic under Sepsis-2 but not under Sepsis-3. (4) Conclusions: This study validates SeptiCyte RAPID for estimating sepsis probability, under both the Sepsis-2 and Sepsis-3 frameworks, for hospitalized patients on their first day of ICU admission.

Causes, Diagnostic Testing, and Treatments Related to Clinical Deterioration Events among High-Risk Ward Patients

OBJECTIVE

Timely intervention for clinically deteriorating ward patients requires that care teams accurately diagnose and treat their underlying medical conditions. However, the most common diagnoses leading to deterioration and the relevant therapies provided are poorly characterized. Therefore, we aimed to determine the diagnoses responsible for clinical deterioration, the relevant diagnostic tests ordered, and the treatments administered among high-risk ward patients using manual chart review.

DESIGN

Multicenter retrospective observational study.

SETTING

Inpatient medical-surgical wards at four health systems from 2006-2020 PATIENTS: Randomly selected patients (1,000 from each health system) with clinical deterioration, defined by reaching the 95th percentile of a validated early warning score, electronic Cardiac Arrest Risk Triage (eCART), were included.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Clinical deterioration was confirmed by a trained reviewer or marked as a false alarm if no deterioration occurred for each patient. For true deterioration events, the condition causing deterioration, relevant diagnostic tests ordered, and treatments provided were collected. Of the 4,000 included patients, 2,484 (62%) had clinical deterioration confirmed by chart review. Sepsis was the most common cause of deterioration (41%; n=1,021), followed by arrhythmia (19%; n=473), while liver failure had the highest in-hospital mortality (41%). The most common diagnostic tests ordered were complete blood counts (47% of events), followed by chest x-rays (42%), and cultures (40%), while the most common medication orders were antimicrobials (46%), followed by fluid boluses (34%), and antiarrhythmics (19%).

CONCLUSIONS

We found that sepsis was the most common cause of deterioration, while liver failure had the highest mortality. Complete blood counts and chest x-rays were the most common diagnostic tests ordered, and antimicrobials and fluid boluses were the most common medication interventions. These results provide important insights for clinical decision-making at the bedside, training of rapid response teams, and the development of institutional treatment pathways for clinical deterioration.

KEY POINTS

Question: What are the most common diagnoses, diagnostic test orders, and treatments for ward patients experiencing clinical deterioration? Findings: In manual chart review of 2,484 encounters with deterioration across four health systems, we found that sepsis was the most common cause of clinical deterioration, followed by arrythmias, while liver failure had the highest mortality. Complete blood counts and chest x-rays were the most common diagnostic test orders, while antimicrobials and fluid boluses were the most common treatments. Meaning: Our results provide new insights into clinical deterioration events, which can inform institutional treatment pathways, rapid response team training, and patient care.

Association of Neighborhood Disadvantage with Short- and Long-Term Outcomes After Pancreatectomy for Pancreatic Ductal Adenocarcinoma

BACKGROUND

While lower socioeconomic status has been shown to correlate with worse outcomes in cancer care, data correlating neighborhood-level metrics with outcomes are scarce. We aim to explore the association between neighborhood disadvantage and both short- and long-term postoperative outcomes in patients undergoing pancreatectomy for pancreatic ductal adenocarcinoma (PDAC).

PATIENTS AND METHODS

We retrospectively analyzed 243 patients who underwent resection for PDAC at a single institution between 1 January 2010 and 15 September 2021. To measure neighborhood disadvantage, the cohort was divided into tertiles by Area Deprivation Index (ADI). Short-term outcomes of interest were minor complications, major complications, unplanned readmission within 30 days, prolonged hospitalization, and delayed gastric emptying (DGE). The long-term outcome of interest was overall survival. Logistic regression was used to test short-term outcomes; Cox proportional hazards models and Kaplan-Meier method were used for long-term outcomes.

RESULTS

The median ADI of the cohort was 49 (IQR 32-64.5). On adjusted analysis, the high-ADI group demonstrated greater odds of suffering a major complication (odds ratio [OR], 2.78; 95% confidence interval [CI], 1.26-6.40; p = 0.01) and of an unplanned readmission (OR, 3.09; 95% CI, 1.16-9.28; p = 0.03) compared with the low-ADI group. There were no significant differences between groups in the odds of minor complications, prolonged hospitalization, or DGE (all p > 0.05). High ADI did not confer an increased hazard of death (p = 0.63).

CONCLUSIONS

We found that worse neighborhood disadvantage is associated with a higher risk of major complication and unplanned readmission after pancreatectomy for PDAC.

Improving model transferability for clinical note section classification models using continued pretraining

OBJECTIVE

The classification of clinical note sections is a critical step before doing more fine-grained natural language processing tasks such as social determinants of health extraction and temporal information extraction. Often, clinical note section classification models that achieve high accuracy for 1 institution experience a large drop of accuracy when transferred to another institution. The objective of this study is to develop methods that classify clinical note sections under the SOAP ("Subjective," "Object," "Assessment," and "Plan") framework with improved transferability.

MATERIALS AND METHODS

We trained the baseline models by fine-tuning BERT-based models, and enhanced their transferability with continued pretraining, including domain-adaptive pretraining and task-adaptive pretraining. We added in-domain annotated samples during fine-tuning and observed model performance over a varying number of annotated sample size. Finally, we quantified the impact of continued pretraining in equivalence of the number of in-domain annotated samples added.

RESULTS

We found continued pretraining improved models only when combined with in-domain annotated samples, improving the F1 score from 0.756 to 0.808, averaged across 3 datasets. This improvement was equivalent to adding 35 in-domain annotated samples.

DISCUSSION

Although considered a straightforward task when performing in-domain, section classification is still a considerably difficult task when performing cross-domain, even using highly sophisticated neural network-based methods.

CONCLUSION

Continued pretraining improved model transferability for cross-domain clinical note section classification in the presence of a small amount of in-domain labeled samples.

Association between mortality and critical events within 48 hours of transfer to the pediatric intensive care unit

Introduction

Critical deterioration in hospitalized children, defined as ward to pediatric intensive care unit (PICU) transfer followed by mechanical ventilation (MV) or vasoactive infusion (VI) within 12 h, has been used as a primary metric to evaluate the effectiveness of clinical interventions or quality improvement initiatives. We explore the association between critical events (CEs), i.e., MV or VI events, within the first 48 h of PICU transfer from the ward or emergency department (ED) and in-hospital mortality.

Methods

We conducted a retrospective study of a cohort of PICU transfers from the ward or the ED at two tertiary-care academic hospitals. We determined the association between mortality and occurrence of CEs within 48 h of PICU transfer after adjusting for age, gender, hospital, and prior comorbidities.

Results

Experiencing a CE within 48 h of PICU transfer was associated with an increased risk of mortality [OR 12.40 (95% CI: 8.12-19.23, P < 0.05)]. The increased risk of mortality was highest in the first 12 h [OR 11.32 (95% CI: 7.51-17.15, P < 0.05)] but persisted in the 12-48 h time interval [OR 2.84 (95% CI: 1.40-5.22, P < 0.05)]. Varying levels of risk were observed when considering ED or ward transfers only, when considering different age groups, and when considering individual 12-h time intervals.

Discussion

We demonstrate that occurrence of a CE within 48 h of PICU transfer was associated with mortality after adjusting for confounders. Studies focusing on the impact of quality improvement efforts may benefit from using CEs within 48 h of PICU transfer as an additional evaluation metric, provided these events could have been influenced by the initiative.

Development and external validation of multimodal postoperative acute kidney injury risk machine learning models

Objectives

To develop and externally validate machine learning models using structured and unstructured electronic health record data to predict postoperative acute kidney injury (AKI) across inpatient settings.

Materials and Methods

Data for adult postoperative admissions to the Loyola University Medical Center (2009-2017) were used for model development and admissions to the University of Wisconsin-Madison (2009-2020) were used for validation. Structured features included demographics, vital signs, laboratory results, and nurse-documented scores. Unstructured text from clinical notes were converted into concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. The primary outcome was the development of Kidney Disease Improvement Global Outcomes stage 2 AKI within 7 days after leaving the operating room. We derived unimodal extreme gradient boosting machines (XGBoost) and elastic net logistic regression (GLMNET) models using structured-only data and multimodal models combining structured data with CUI features. Model comparison was performed using the receiver operating characteristic curve (AUROC), with Delong's test for statistical differences.

Results

The study cohort included 138 389 adult patient admissions (mean [SD] age 58 [16] years; 11 506 [8%] African-American; and 70 826 [51%] female) across the 2 sites. Of those, 2959 (2.1%) developed stage 2 AKI or higher. Across all data types, XGBoost outperformed GLMNET (mean AUROC 0.81 [95% confidence interval (CI), 0.80-0.82] vs 0.78 [95% CI, 0.77-0.79]). The multimodal XGBoost model incorporating CUIs parameterized as term frequency-inverse document frequency (TF-IDF) showed the highest discrimination performance (AUROC 0.82 [95% CI, 0.81-0.83]) over unimodal models (AUROC 0.79 [95% CI, 0.78-0.80]).

Discussion

A multimodality approach with structured data and TF-IDF weighting of CUIs increased model performance over structured data-only models.

Conclusion

These findings highlight the predictive power of CUIs when merged with structured data for clinical prediction models, which may improve the detection of postoperative AKI.

Creation of a data commons for substance misuse related health research through privacy-preserving patient record linkage between hospitals and state agencies

Objectives

Substance misuse is a complex and heterogeneous set of conditions associated with high mortality and regional/demographic variations. Existing data systems are siloed and have been ineffective in curtailing the substance misuse epidemic. Therefore, we aimed to build a novel informatics platform, the Substance Misuse Data Commons (SMDC), by integrating multiple data modalities to provide a unified record of information crucial to improving outcomes in substance misuse patients.

Materials and Methods

The SMDC was created by linking electronic health record (EHR) data from adult cases of substance (alcohol, opioid, nonopioid drug) misuse at the University of Wisconsin hospitals to socioeconomic and state agency data. To ensure private and secure data exchange, Privacy-Preserving Record Linkage (PPRL) and Honest Broker services were utilized. The overlap in mortality reporting among the EHR, state Vital Statistics, and a commercial national data source was assessed.

Results

The SMDC included data from 36 522 patients experiencing 62 594 healthcare encounters. Over half of patients were linked to the statewide ambulance database and prescription drug monitoring program. Chronic diseases accounted for most underlying causes of death, while drug-related overdoses constituted 8%. Our analysis of mortality revealed a 49.1% overlap across the 3 data sources. Nonoverlapping deaths were associated with poor socioeconomic indicators.

Discussion

Through PPRL, the SMDC enabled the longitudinal integration of multimodal data. Combining death data from local, state, and national sources enhanced mortality tracking and exposed disparities.

Conclusion

The SMDC provides a comprehensive resource for clinical providers and policymakers to inform interventions targeting substance misuse-related hospitalizations, overdoses, and death.

Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning

Generative artificial intelligence (AI) is a promising direction for augmenting clinical diagnostic decision support and reducing diagnostic errors, a leading contributor to medical errors. To further the development of clinical AI systems, the Diagnostic Reasoning Benchmark (DR.BENCH) was introduced as a comprehensive generative AI framework, comprised of six tasks representing key components in clinical reasoning. We present a comparative analysis of in-domain versus out-of-domain language models as well as multi-task versus single task training with a focus on the problem summarization task in DR.BENCH (Gao et al., 2023). We demonstrate that a multi-task, clinically-trained language model outperforms its general domain counterpart by a large margin, establishing a new state-of-the-art performance, with a ROUGE-L score of 28.55. This research underscores the value of domain-specific training for optimizing clinical diagnostic reasoning tasks.

Improving the Transferability of Clinical Note Section Classification Models with BERT and Large Language Model Ensembles

Text in electronic health records is organized into sections, and classifying those sections into section categories is useful for downstream tasks. In this work, we attempt to improve the transferability of section classification models by combining the dataset-specific knowledge in supervised learning models with the world knowledge inside large language models (LLMs). Surprisingly, we find that zero-shot LLMs out-perform supervised BERT-based models applied to out-of-domain data. We also find that their strengths are synergistic, so that a simple ensemble technique leads to additional performance gains.

Playing in the clinical decision support sandbox: tools and training for all

Objectives

Introduce the CDS-Sandbox, a cloud-based virtual machine created to facilitate Clinical Decision Support (CDS) developers and implementers in the use of FHIR- and CQL-based open-source tools and technologies for building and testing CDS artifacts.

Materials and Methods

The CDS-Sandbox includes components that enable workflows for authoring and testing CDS artifacts. Two workshops at the 2020 and 2021 AMIA Annual Symposia were conducted to demonstrate the use of the open-source CDS tools.

Results

The CDS-Sandbox successfully integrated the use of open-source CDS tools. Both workshops were well attended. Participants demonstrated use and understanding of the workshop materials and provided positive feedback after the workshops.

Discussion

The CDS-Sandbox and publicly available tutorial materials facilitated an understanding of the leading-edge open-source CDS infrastructure components.

Conclusion

The CDS-Sandbox supports integrated use of the key CDS open-source tools that may be used to introduce CDS concepts and practice to the clinical informatics community.

Overview of the Problem List Summarization (ProbSum) 2023 Shared Task on Summarizing Patients' Active Diagnoses and Problems from Electronic Health Record Progress Notes

The BioNLP Workshop 2023 initiated the launch of a shared task on Problem List Summarization (ProbSum) in January 2023. The aim of this shared task is to attract future research efforts in building NLP models for real-world diagnostic decision support applications, where a system generating relevant and accurate diagnoses will augment the healthcare providers' decision-making process and improve the quality of care for patients. The goal for participants is to develop models that generated a list of diagnoses and problems using input from the daily care notes collected from the hospitalization of critically ill patients. Eight teams submitted their final systems to the shared task leaderboard. In this paper, we describe the tasks, datasets, evaluation metrics, and baseline systems. Additionally, the techniques and results of the evaluation of the different approaches tried by the participating teams are summarized.

Progress Note Understanding - Assessment and Plan Reasoning: Overview of the 2022 N2C2 Track 3 shared task

Daily progress notes are a common note type in the electronic health record (EHR) where healthcare providers document the patient's daily progress and treatment plans. The EHR is designed to document all the care provided to patients, but it also enables note bloat with extraneous information that distracts from the diagnoses and treatment plans. Applications of natural language processing (NLP) in the EHR is a growing field with the majority of methods in information extraction. Few tasks use NLP methods for downstream diagnostic decision support. We introduced the 2022 National NLP Clinical Challenge (N2C2) Track 3: Progress Note Understanding - Assessment and Plan Reasoning as one step towards a new suite of tasks. The Assessment and Plan Reasoning task focuses on the most critical components of progress notes, Assessment and Plan subsections where health problems and diagnoses are contained. The goal of the task was to develop and evaluate NLP systems that automatically predict causal relations between the overall status of the patient contained in the Assessment section and its relation to each component of the Plan section which contains the diagnoses and treatment plans. The goal of the task was to identify and prioritize diagnoses as the first steps in diagnostic decision support to find the most relevant information in long documents like daily progress notes. We present the results of the 2022 N2C2 Track 3 and provide a description of the data, evaluation, participation and system performance.

Design and rationale of the CHILL phase II trial of hypothermia and neuromuscular blockade for acute respiratory distress syndrome

The Cooling to Help Injured Lungs (CHILL) trial is an open label, two group, parallel design multicenter, randomized phase IIB clinical trial assessing the efficacy and safety of targeted temperature management with combined external cooling and neuromuscular blockade to block shivering in patients with early moderate-severe acute respiratory distress syndrome (ARDS). This report provides the background and rationale for the clinical trial and outlines the methods using the Consolidated Standards of Reporting Trials guidelines. Key design challenges include: [1] protocolizing important co-interventions; [2] incorporation of patients with COVID-19 as the cause of ARDS; [3] inability to blind the investigators; and [4] ability to obtain timely informed consent from patients or legally authorized representatives early in the disease process. Results of the Reevaluation of Systemic Early Neuromuscular Blockade (ROSE) trial informed the decision to mandate sedation and neuromuscular blockade only in the group assigned to therapeutic hypothermia and proceed without this mandate in the control group assigned to a usual temperature management protocol. Previous trials conducted in National Heart, Lung, and Blood Institute ARDS Clinical Trials (ARDSNet) and Prevention and Early Treatment of Acute Lung Injury (PETAL) Networks informed ventilator management, ventilation liberation and fluid management protocols. Since ARDS due to COVID-19 is a common cause of ARDS during pandemic surges and shares many features with ARDS from other causes, patients with ARDS due to COVID-19 are included. Finally, a stepwise approach to obtaining informed consent prior to documenting critical hypoxemia was adopted to facilitate enrollment and reduce the number of candidates excluded because eligibility time window expiration.

Improving Model Transferability for Clinical Note Section Classification Models Using Continued Pretraining

Objective

The classification of clinical note sections is a critical step before doing more fine-grained natural language processing tasks such as social determinants of health extraction and temporal information extraction. Often, clinical note section classification models that achieve high accuracy for one institution experience a large drop of accuracy when transferred to another institution. The objective of this study is to develop methods that classify clinical note sections under the SOAP ("Subjective", "Object", "Assessment" and "Plan") framework with improved transferability.

Materials and methods

We trained the baseline models by fine-tuning BERT-based models, and enhanced their transferability with continued pretraining, including domain adaptive pretraining (DAPT) and task adaptive pretraining (TAPT). We added out-of-domain annotated samples during fine-tuning and observed model performance over a varying number of annotated sample size. Finally, we quantified the impact of continued pretraining in equivalence of the number of in-domain annotated samples added.

Results

We found continued pretraining improved models only when combined with in-domain annotated samples, improving the F1 score from 0.756 to 0.808, averaged across three datasets. This improvement was equivalent to adding 50.2 in-domain annotated samples.

Discussion

Although considered a straightforward task when performing in-domain, section classification is still a considerably difficult task when performing cross-domain, even using highly sophisticated neural network-based methods.

Conclusion

Continued pretraining improved model transferability for cross-domain clinical note section classification in the presence of a small amount of in-domain labeled samples.

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults

BACKGROUND

The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery.

OBJECTIVE

We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool.

METHODS

The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan.

RESULTS

The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR.

CONCLUSIONS

The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.

Accuracy of models to prognosticate survival after surgery for pancreatic cancer in the era of neoadjuvant therapy

BACKGROUND

Outcomes for pancreatic adenocarcinoma (PDAC) remain difficult to prognosticate. Multiple models attempt to predict survival following the resection of PDAC, but their utility in the neoadjuvant population is unknown. We aimed to assess their accuracy among patients that received neoadjuvant chemotherapy (NAC).

METHODS

We performed a multi-institutional retrospective analysis of patients who received NAC and underwent resection of PDAC. Two prognostic systems were evaluated: the Memorial Sloan Kettering Cancer Center Pancreatic Adenocarcinoma Nomogram (MSKCCPAN) and the American Joint Committee on Cancer (AJCC) staging system. Discrimination between predicted and actual disease-specific survival was assessed using the Uno C-statistic and Kaplan-Meier method. Calibration of the MSKCCPAN was assessed using the Brier score.

RESULTS

A total of 448 patients were included. There were 232 (51.8%) females, and the mean age was 64.1 years (±9.5). Most had AJCC Stage I or II disease (77.7%). For the MSKCCPAN, the Uno C-statistic at 12-, 24-, and 36-month time points was 0.62, 0.63, and 0.62, respectively. The AJCC system demonstrated similarly mediocre discrimination. The Brier score for the MSKCCPAN was 0.15 at 12 months, 0.26 at 24 months, and 0.30 at 36 months, demonstrating modest calibration.

CONCLUSIONS

Current survival prediction models and staging systems for patients with PDAC undergoing resection after NAC have limited accuracy.

Association between Substance Misuse and Outcomes in Critically III Patients with Pneumonia

Rationale: In patients with pneumonia requiring intensive care unit (ICU) admission, alcohol misuse is associated with increased mortality, but the relationship between other commonly misused substances and mortality is unknown. Objectives: We sought to establish whether alcohol misuse, cannabis misuse, opioid misuse, stimulant misuse, or misuse of more than one of these substances was associated with differences in mortality among ICU patients with pneumonia. Methods: This was a retrospective cohort study of hospitals participating in the Premier Healthcare Database between 2010 and 2017. Patients were included if they had a primary or secondary diagnosis of pneumonia and received antibiotics or antivirals within 1 day of admission. Substance misuse related to alcohol, cannabis, stimulants, and opioids, or more than one substance, were identified from the International Classification of Diseases (Ninth and Tenth Editions). The associations between substance misuse and in-hospital mortality were the primary outcomes of interest. Secondary outcomes included the measured associations between substance misuse disorders and mechanical ventilation, as well as vasopressor and continuous paralytic administration. Analyses were conducted with multivariable mixed-effects logistic regression modeling adjusting for age, comorbidities, and hospital characteristics. Results: A total of 167,095 ICU patients met inclusion criteria for pneumonia. Misuse of alcohol was present in 5.0%, cannabis misuse in 0.6%, opioid misuse in 1.5%, stimulant misuse in 0.6%, and misuse of more than one substance in 1.2%. No evidence of substance misuse was found in 91.1% of patients. In unadjusted analyses, alcohol misuse was associated with increased in-hospital mortality (odds ratio [OR], 1.12; 95% confidence interval [CI], 1.06-1.19), whereas opioid misuse was associated with decreased in-hospital mortality (OR, 0.46; 95% CI, 0.39-0.53) compared with no substance misuse. These findings persisted in adjusted analyses. Although cannabis, stimulant, and more than one substance misuse (a majority of which were alcohol in combination with another substance) were associated with lower odds for in-hospital mortality in unadjusted analyses, these relationships were not consistently present after adjustment. Conclusions: In this study of ICU patients hospitalized with severe pneumonia, substance misuse subtypes were associated with different effects on mortality. Although administrative data can provide epidemiologic insight regarding substance misuse and pneumonia outcomes, biases inherent to these data should be considered when interpreting results.

Perceptions and Experiences With Evidence-based Treatments Among People Who Use Opioids

OBJECTIVES

Even where treatment is available, people who use drugs (PWUD) may not seek help. Few published studies examine beliefs, experiences, and perceptions of evidence-based treatment among PWUD who are not actively engaged in care. This study aimed to explore the experiences of PWUD in considering or accessing treatment and gauge receptiveness to low-threshold treatment models.

METHODS

A purposeful sample of participants actively using opioids and with previous interest in or experience with treatment was recruited from a harm reduction program in Chicago. Semistructured interviews were conducted to explore key phenomena while allowing for unanticipated themes. The instrument included questions about historical drug use, treatment experience, and perceptions of how to improve treatment access and services. Private interviews were audio recorded, transcribed, and double coded by 2 analysts. Queries of coded data were analyzed using issue-focused analysis to identify themes.

RESULTS

The sample (N = 40) approximated groups at highest risk of fatal overdose in Chicago, with more than 80% between the ages of 45 to 64 years, 65% African American, and 62% male identified. The majority had prior treatment experience, although all resumed use after completing or leaving treatment. The most prevalent barriers to treatment included structural barriers related to social determinants, lack of readiness for abstinence, burdensome intake procedures, and regulatory/programmatic requirements. Most participants expressed interest in low-threshold treatment.

CONCLUSIONS

Existing treatment barriers may be addressed by shifting to lower-threshold intake processes and/or outreach-based delivery of opioid agonist treatment. Engaging PWUD in efforts to create lower-threshold treatment programs is necessary to ensure that needs are met.

DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing

The meaningful use of electronic health records (EHR) continues to progress in the digital era with clinical decision support systems augmented by artificial intelligence. A priority in improving provider experience is to overcome information overload and reduce the cognitive burden so fewer medical errors and cognitive biases are introduced during patient care. One major type of medical error is diagnostic error due to systematic or predictable errors in judgement that rely on heuristics. The potential for clinical natural language processing (cNLP) to model diagnostic reasoning in humans with forward reasoning from data to diagnosis and potentially reduce cognitive burden and medical error has not been investigated. Existing tasks to advance the science in cNLP have largely focused on information extraction and named entity recognition through classification tasks. We introduce a novel suite of tasks coined as Diagnostic Reasoning Benchmarks, Dr.Bench, as a new benchmark for developing and evaluating cNLP models with clinical diagnostic reasoning ability. The suite includes six tasks from ten publicly available datasets addressing clinical text understanding, medical knowledge reasoning, and diagnosis generation. DR.BENCH is the first clinical suite of tasks designed to be a natural language generation framework to evaluate pre-trained language models for diagnostic reasoning. The goal of DR. BENCH is to advance the science in cNLP to support downstream applications in computerized diagnostic decision support and improve the efficiency and accuracy of healthcare providers during patient care. We fine-tune and evaluate the state-of-the-art generative models on DR.BENCH. Experiments show that with domain adaptation pre-training on medical knowledge, the model demonstrated opportunities for improvement when evaluated in DR. BENCH. We share DR. BENCH as a publicly available GitLab repository with a systematic approach to load and evaluate models for the cNLP community. We also discuss the carbon footprint produced during the experiments and encourage future work on DR.BENCH to report the carbon footprint.

Clinical Trial Design Innovations for Precision Medicine in Asthma

Severe asthma is a spectrum disorder with numerous subsets, many of which are defined by clinical history and a general predisposition for T2 inflammation. Most of the approved therapies for severe asthma have required clinical trial designs with population enrichment for exacerbation frequency and/or elevation of blood eosinophils. Moving beyond this framework will require trial designs that increase efficiency for studying nondominant subsets and continue to improve upon biomarker signatures. In addition to reviewing the current literature on biomarker-informed trials for severe asthma, this chapter will also review the advantages of master protocols and adaptive design methods for establishing the efficacy of new interventions in prospectively defined subsets of patients. The incorporation of methods that allow for data collection outside of traditional study visits at academic centers, called remote decentralized trial design, is a growing trend that may increase diversity in study participation and allow for enhanced resiliency during the COVID-19 pandemic. Finally, reaching the goals of precision medicine in asthma will require increased emphasis on effectiveness studies. Recent advances in real-world data utilization from electronic health records are also discussed with a view toward pragmatic trial designs that could also incorporate the evaluation of biomarker signatures.

The Evaluation of a Clinical Decision Support Tool Using Natural Language Processing to Screen Hospitalized Adults for Unhealthy Substance Use: Protocol for a Quasi-Experimental Design

BACKGROUND

Automated and data-driven methods for screening using natural language processing (NLP) and machine learning may replace resource-intensive manual approaches in the usual care of patients hospitalized with conditions related to unhealthy substance use. The rigorous evaluation of tools that use artificial intelligence (AI) is necessary to demonstrate effectiveness before system-wide implementation. An NLP tool to use routinely collected data in the electronic health record was previously validated for diagnostic accuracy in a retrospective study for screening unhealthy substance use. Our next step is a noninferiority design incorporated into a research protocol for clinical implementation with prospective evaluation of clinical effectiveness in a large health system.

OBJECTIVE

This study aims to provide a study protocol to evaluate health outcomes and the costs and benefits of an AI-driven automated screener compared to manual human screening for unhealthy substance use.

METHODS

A pre-post design is proposed to evaluate 12 months of manual screening followed by 12 months of automated screening across surgical and medical wards at a single medical center. The preintervention period consists of usual care with manual screening by nurses and social workers and referrals to a multidisciplinary Substance Use Intervention Team (SUIT). Facilitated by a NLP pipeline in the postintervention period, clinical notes from the first 24 hours of hospitalization will be processed and scored by a machine learning model, and the SUIT will be similarly alerted to patients who flagged positive for substance misuse. Flowsheets within the electronic health record have been updated to capture rates of interventions for the primary outcome (brief intervention/motivational interviewing, medication-assisted treatment, naloxone dispensing, and referral to outpatient care). Effectiveness in terms of patient outcomes will be determined by noninferior rates of interventions (primary outcome), as well as rates of readmission within 6 months, average time to consult, and discharge rates against medical advice (secondary outcomes) in the postintervention period by a SUIT compared to the preintervention period. A separate analysis will be performed to assess the costs and benefits to the health system by using automated screening. Changes from the pre- to postintervention period will be assessed in covariate-adjusted generalized linear mixed-effects models.

RESULTS

The study will begin in September 2022. Monthly data monitoring and Data Safety Monitoring Board reporting are scheduled every 6 months throughout the study period. We anticipate reporting final results by June 2025.

CONCLUSIONS

The use of augmented intelligence for clinical decision support is growing with an increasing number of AI tools. We provide a research protocol for prospective evaluation of an automated NLP system for screening unhealthy substance use using a noninferiority design to demonstrate comprehensive screening that may be as effective as manual screening but less costly via automated solutions.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03833804; https://clinicaltrials.gov/ct2/show/NCT03833804.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/42971.

Machine Learning Techniques to Explore Clinical Presentations of COVID-19 Severity and to Test the Association With Unhealthy Opioid Use: Retrospective Cross-sectional Cohort Study

BACKGROUND

The COVID-19 pandemic has exacerbated health inequities in the United States. People with unhealthy opioid use (UOU) may face disproportionate challenges with COVID-19 precautions, and the pandemic has disrupted access to opioids and UOU treatments. UOU impairs the immunological, cardiovascular, pulmonary, renal, and neurological systems and may increase severity of outcomes for COVID-19.

OBJECTIVE

We applied machine learning techniques to explore clinical presentations of hospitalized patients with UOU and COVID-19 and to test the association between UOU and COVID-19 disease severity.

METHODS

This retrospective, cross-sectional cohort study was conducted based on data from 4110 electronic health record patient encounters at an academic health center in Chicago between January 1, 2020, and December 31, 2020. The inclusion criterion was an unplanned admission of a patient aged ≥18 years; encounters were counted as COVID-19-positive if there was a positive test for COVID-19 or 2 COVID-19 International Classification of Disease, Tenth Revision codes. Using a predefined cutoff with optimal sensitivity and specificity to identify UOU, we ran a machine learning UOU classifier on the data for patients with COVID-19 to estimate the subcohort of patients with UOU. Topic modeling was used to explore and compare the clinical presentations documented for 2 subgroups: encounters with UOU and COVID-19 and those with no UOU and COVID-19. Mixed effects logistic regression accounted for multiple encounters for some patients and tested the association between UOU and COVID-19 outcome severity. Severity was measured with 3 utilization metrics: low-severity unplanned admission, medium-severity unplanned admission and receiving mechanical ventilation, and high-severity unplanned admission with in-hospital death. All models controlled for age, sex, race/ethnicity, insurance status, and BMI.

RESULTS

Topic modeling yielded 10 topics per subgroup and highlighted unique comorbidities associated with UOU and COVID-19 (eg, HIV) and no UOU and COVID-19 (eg, diabetes). In the regression analysis, each incremental increase in the classifier's predicted probability of UOU was associated with 1.16 higher odds of COVID-19 outcome severity (odds ratio 1.16, 95% CI 1.04-1.29; P=.009).

CONCLUSIONS

Among patients hospitalized with COVID-19, UOU is an independent risk factor associated with greater outcome severity, including in-hospital death. Social determinants of health and opioid-related overdose are unique comorbidities in the clinical presentation of the UOU patient subgroup. Additional research is needed on the role of COVID-19 therapeutics and inpatient management of acute COVID-19 pneumonia for patients with UOU. Further research is needed to test associations between expanded evidence-based harm reduction strategies for UOU and vaccination rates, hospitalizations, and risks for overdose and death among people with UOU and COVID-19. Machine learning techniques may offer more exhaustive means for cohort discovery and a novel mixed methods approach to population health.

Summarizing Patients' Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models

Automatically summarizing patients' main problems from daily progress notes using natural language processing methods helps to battle against information and cognitive overload in hospital settings and potentially assists providers with computerized diagnostic decision support. Problem list summarization requires a model to understand, abstract, and generate clinical documentation. In this work, we propose a new NLP task that aims to generate a list of problems in a patient's daily care plan using input from the provider's progress notes during hospitalization. We investigate the performance of T5 and BART, two state-of-the-art seq2seq transformer architectures, in solving this problem. We provide a corpus built on top of progress notes from publicly available electronic health record progress notes in the Medical Information Mart for Intensive Care (MIMIC)-III. T5 and BART are trained on general domain text, and we experiment with a data augmentation method and a domain adaptation pre-training method to increase exposure to medical vocabulary and knowledge. Evaluation methods include ROUGE, BERTScore, cosine similarity on sentence embedding, and F-score on medical concepts. Results show that T5 with domain adaptive pre-training achieves significant performance gains compared to a rule-based system and general domain pre-trained language models, indicating a promising direction for tackling the problem summarization task.

Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes

One of the key challenges in successful deployment and meaningful adoption of AI in healthcare is health system-level governance of AI applications. Such governance is critical not only for patient safety and accountability by a health system, but to foster clinician trust to improve adoption and facilitate meaningful health outcomes. In this case study, we describe the development of such a governance structure at University of Wisconsin Health (UWH) that provides oversight of AI applications from assessment of validity and user acceptability through safe deployment with continuous monitoring for effectiveness. Our structure leverages a multi-disciplinary steering committee along with project specific sub-committees. Members of the committee formulate a multi-stakeholder perspective spanning informatics, data science, clinical operations, ethics, and equity. Our structure includes guiding principles that provide tangible parameters for endorsement of both initial deployment and ongoing usage of AI applications. The committee is tasked with ensuring principles of interpretability, accuracy, and fairness across all applications. To operationalize these principles, we provide a value stream to apply the principles of AI governance at different stages of clinical implementation. This structure has enabled effective clinical adoption of AI applications. Effective governance has provided several outcomes: (1) a clear and institutional structure for oversight and endorsement; (2) a path towards successful deployment that encompasses technologic, clinical, and operational, considerations; (3) a process for ongoing monitoring to ensure the solution remains acceptable as clinical practice and disease prevalence evolve; (4) incorporation of guidelines for the ethical and equitable use of AI applications.

A scoping review of publicly available language tasks in clinical natural language processing

OBJECTIVE

To provide a scoping review of papers on clinical natural language processing (NLP) shared tasks that use publicly available electronic health record data from a cohort of patients.

MATERIALS AND METHODS

We searched 6 databases, including biomedical research and computer science literature databases. A round of title/abstract screening and full-text screening were conducted by 2 reviewers. Our method followed the PRISMA-ScR guidelines.

RESULTS

A total of 35 papers with 48 clinical NLP tasks met inclusion criteria between 2007 and 2021. We categorized the tasks by the type of NLP problems, including named entity recognition, summarization, and other NLP tasks. Some tasks were introduced as potential clinical decision support applications, such as substance abuse detection, and phenotyping. We summarized the tasks by publication venue and dataset type.

DISCUSSION

The breadth of clinical NLP tasks continues to grow as the field of NLP evolves with advancements in language systems. However, gaps exist with divergent interests between the general domain NLP community and the clinical informatics community for task motivation and design, and in generalizability of the data sources. We also identified issues in data preparation.

CONCLUSION

The existing clinical NLP tasks cover a wide range of topics and the field is expected to grow and attract more attention from both general domain NLP and clinical informatics community. We encourage future work to incorporate multidisciplinary collaboration, reporting transparency, and standardization in data preparation. We provide a listing of all the shared task papers and datasets from this review in a GitLab repository.

Identifying infected patients using semi-supervised and transfer learning

Objectives

Early identification of infection improves outcomes, but developing models for early identification requires determining infection status with manual chart review, limiting sample size. Therefore, we aimed to compare semi-supervised and transfer learning algorithms with algorithms based solely on manual chart review for identifying infection in hospitalized patients.

Materials and Methods

This multicenter retrospective study of admissions to 6 hospitals included “gold-standard” labels of infection from manual chart review and “silver-standard” labels from nonchart-reviewed patients using the Sepsis-3 infection criteria based on antibiotic and culture orders. “Gold-standard” labeled admissions were randomly allocated to training (70%) and testing (30%) datasets. Using patient characteristics, vital signs, and laboratory data from the first 24 hours of admission, we derived deep learning and non-deep learning models using transfer learning and semi-supervised methods. Performance was compared in the gold-standard test set using discrimination and calibration metrics.

Results

The study comprised 432 965 admissions, of which 2724 underwent chart review. In the test set, deep learning and non-deep learning approaches had similar discrimination (area under the receiver operating characteristic curve of 0.82). Semi-supervised and transfer learning approaches did not improve discrimination over models fit using only silver- or gold-standard data. Transfer learning had the best calibration (unreliability index P value: .997, Brier score: 0.173), followed by self-learning gradient boosted machine (P value: .67, Brier score: 0.170).

Discussion

Deep learning and non-deep learning models performed similarly for identifying infection, as did models developed using Sepsis-3 and manual chart review labels.

Conclusion

In a multicenter study of almost 3000 chart-reviewed patients, semi-supervised and transfer learning models showed similar performance for model discrimination as baseline XGBoost, while transfer learning improved calibration.

The Identification of Subphenotypes and Associations with Health Outcomes in Patients with Opioid-Related Emergency Department Encounters Using Latent Class Analysis

The emergency department (ED) is a critical setting for the treatment of patients with opioid misuse. Detecting relevant clinical profiles allows for tailored treatment approaches. We sought to identify and characterize subphenotypes of ED patients with opioid-related encounters. A latent class analysis was conducted using 14,057,302 opioid-related encounters from 2016 through 2017 using the National Emergency Department Sample (NEDS), the largest all-payer ED database in the United States. The optimal model was determined by face validity and information criteria-based metrics. A three-step approach assessed class structure, assigned individuals to classes, and examined characteristics between classes. Class associations were determined for hospitalization, in-hospital death, and ED charges. The final five-class model consisted of the following subphenotypes: Chronic pain (class 1); Alcohol use (class 2); Depression and pain (class 3); Psychosis, liver disease, and polysubstance use (class 4); and Pregnancy (class 5). Using class 1 as the reference, the greatest odds for hospitalization occurred in classes 3 and 4 (Ors 5.24 and 5.33, p < 0.001) and for in-hospital death in class 4 (OR 3.44, p < 0.001). Median ED charges ranged from USD 2177 (class 1) to USD 2881 (class 4). These subphenotypes provide a basis for examining patient-tailored approaches for this patient population.

Hierarchical Annotation for Building A Suite of Clinical Natural Language Processing Tasks: Progress Note Understanding

Applying methods in natural language processing on electronic health records (EHR) data is a growing field. Existing corpus and annotation focus on modeling textual features and relation prediction. However, there is a paucity of annotated corpus built to model clinical diagnostic thinking, a process involving text understanding, domain knowledge abstraction and reasoning. This work introduces a hierarchical annotation schema with three stages to address clinical text understanding, clinical reasoning, and summarization. We created an annotated corpus based on an extensive collection of publicly available daily progress notes, a type of EHR documentation that is collected in time series in a problem-oriented format. The conventional format for a progress note follows a Subjective, Objective, Assessment and Plan heading (SOAP). We also define a new suite of tasks, Progress Note Understanding, with three tasks utilizing the three annotation stages. The novel suite of tasks was designed to train and evaluate future NLP models for clinical text understanding, clinical knowledge representation, inference, and summarization.

Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study

Background

Substance misuse is a heterogeneous and complex set of behavioural conditions that are highly prevalent in hospital settings and frequently co-occur. Few hospital-wide solutions exist to comprehensively and reliably identify these conditions to prioritise care and guide treatment. The aim of this study was to apply natural language processing (NLP) to clinical notes collected in the electronic health record (EHR) to accurately screen for substance misuse.

Methods

The model was trained and developed on a reference dataset derived from a hospital-wide programme at Rush University Medical Center (RUMC), Chicago, IL, USA, that used structured diagnostic interviews to manually screen admitted patients over 27 months (between Oct 1, 2017, and Dec 31, 2019; n=54 915). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 h of notes in the EHR were mapped to standardised medical vocabulary and fed into single-label, multilabel, and multilabel with auxillary-task neural network models. Temporal validation of the model was done using data from the subsequent 12 months on a subset of RUMC patients (n=16 917). External validation was done using data from Loyola University Medical Center, Chicago, IL, USA between Jan 1, 2007, and Sept 30, 2017 (n=1991 adult patients). The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating characteristic curve (AUROC). Calibration slope and intercept were measured with the unreliability index. Bias assessments were performed across demographic subgroups.

Findings

The model was trained on a cohort that had 3·5% misuse (n=1 921) with any type of substance. 220 (11%) of 1921 patients with substance misuse had more than one type of misuse. The multilabel convolutional neural network classifier had a mean AUROC of 0·97 (95% CI 0·96–0·98) during temporal validation for all types of substance misuse. The model was well calibrated and showed good face validity with model features containing explicit mentions of aberrant drug-taking behaviour. A false-negative rate of 0·18–0·19 and a false-positive rate of 0·03 between non-Hispanic Black and non-Hispanic White groups occurred. In external validation, the AUROCs for alcohol and opioid misuse were 0·88 (95% CI 0·86–0·90) and 0·94 (0·92–0·95), respectively.

Interpretation

We developed a novel and accurate approach to leveraging the first 24 h of EHR notes for screening multiple types of substance misuse.

Funding

National Institute On Drug Abuse, National Institutes of Health.

Metabolomic Associations of Asthma in the Hispanic Community Health Study/Study of Latinos

Asthma disproportionally affects Hispanic and/or Latino backgrounds; however, the relation between circulating metabolites and asthma remains unclear. We conducted a cross-sectional study associating 640 individual serum metabolites, as well as twelve metabolite modules, with asthma in 3347 Hispanic/Latino background participants (514 asthmatics, 15.36%) from the Hispanic/Latino Community Health Study/Study of Latinos. Using survey logistic regression, per standard deviation (SD) increase in 1-arachidonoyl-GPA (20:4) was significantly associated with 32% high odds of asthma after accounting for clinical risk factors (p = 6.27 × 10−5), and per SD of the green module, constructed using weighted gene co-expression network, was suggestively associated with 25% high odds of asthma (p = 0.006). In the stratified analyses by sex and Hispanic and/or Latino backgrounds, the effect of 1-arachidonoyl-GPA (20:4) and the green module was predominantly observed in women (OR = 1.24 and 1.37, p < 0.001) and people of Cuban and Puerto-Rican backgrounds (OR = 1.25 and 1.27, p < 0.01). Mutations in Fatty Acid Desaturase 2 (FADS2) affected the levels of 1-arachidonoyl-GPA (20:4), and Mendelian Randomization analyses revealed that high genetically regulated 1-arachidonoyl-GPA (20:4) levels were associated with increased odds of asthma (p < 0.001). The findings reinforce a molecular basis for asthma etiology, and the potential causal effect of 1-arachidonoyl-GPA (20:4) on asthma provides an opportunity for future intervention.

External validation of a machine learning classifier to identify unhealthy alcohol use in hospitalized patients

BACKGROUND AND AIMS

Unhealthy alcohol use (UAU) is one of the leading causes of global morbidity. A machine learning approach to alcohol screening could accelerate best practices when integrated into electronic health record (EHR) systems. This study aimed to validate externally a natural language processing (NLP) classifier developed at an independent medical center.

DESIGN

Retrospective cohort study.

SETTING

The site for validation was a midwestern United States tertiary-care, urban medical center that has an inpatient structured universal screening model for unhealthy substance use and an active addiction consult service.

PARTICIPANTS/CASES

Unplanned admissions of adult patients between October 23, 2017 and December 31, 2019, with EHR documentation of manual alcohol screening were included in the cohort (n = 57 605).

MEASUREMENTS

The Alcohol Use Disorders Identification Test (AUDIT) served as the reference standard. AUDIT scores ≥5 for females and ≥8 for males served as cases for UAU. To examine error in manual screening or under-reporting, a post hoc error analysis was conducted, reviewing discordance between the NLP classifier and AUDIT-derived reference. All clinical notes excluding the manual screening and AUDIT documentation from the EHR were included in the NLP analysis.

FINDINGS

Using clinical notes from the first 24 hours of each encounter, the NLP classifier demonstrated an area under the receiver operating characteristic curve (AUCROC) and precision-recall area under the curve (PRAUC) of 0.91 (95% CI = 0.89-0.92) and 0.56 (95% CI = 0.53-0.60), respectively. At the optimal cut point of 0.5, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 0.66 (95% CI = 0.62-0.69), 0.98 (95% CI = 0.98-0.98), 0.35 (95% CI = 0.33-0.38), and 1.0 (95% CI = 1.0-1.0), respectively.

CONCLUSIONS

External validation of a publicly available alcohol misuse classifier demonstrates adequate sensitivity and specificity for routine clinical use as an automated screening tool for identifying at-risk patients.

Bias Assessment and Correction in Machine Learning Algorithms: A Use-Case in a Natural Language Processing Algorithm to Identify Hospitalized Patients with Unhealthy Alcohol Use

Unhealthy alcohol use represents a major economic burden and cause of morbidity and mortality in the United States. Implementation of interventions for unhealthy alcohol use depends on the availability and accuracy of screening tools. Our group previously applied methods in natural language processing and machine learning to build a classifier for unhealthy alcohol use. In this study, we sought to evaluate and address bias through the use-case of our classifier. We demonstrated the presence of biased unhealthy alcohol use risk underestimation among Hispanic compared to Non-Hispanic White trauma inpatients, 18- to 44-year-old compared to 45 years and older medical/surgical inpatients, and Non-Hispanic Black compared to Non-Hispanic White medical/surgical inpatients. We further showed that intercept, slope, and concurrent intercept and slope recalibration resulted in minimal or no improvements in bias-indicating metrics within these subgroups. Our results exemplify the importance of integrating bias assessment early into the classifier development pipeline.

The Addition of United States Census-Tract Data Does Not Improve the Prediction of Substance Misuse

Predictors from the structured data in the electronic health record (EHR) have previously been used for case-identification in substance misuse. We aim to examine the added benefit from census-tract data, a proxy for socioeconomic status, to improve identification. A cohort of 186,611 hospitalizations was derived between 2007 and 2017. Reference labels included alcohol misuse only, opioid misuse only, and both alcohol and opioid misuse. Baseline models were created using 24 EHR variables, and enhanced models were created with the addition of 48 census-tract variables from the United States American Community Survey. The absolute net reclassification index (NRI) was applied to measure the benefit in adding census-tract variables to baseline models. The baseline models already had good calibration and discrimination. Adding census-tract variables provided negligible improvement to sensitivity and specificity and NRI was less than 1% across substance groups. Our results show the census-tract added minimal value to prediction models.

A Natural Language Processing Pipeline to Identify COVID-19 Outbreaks from Contact Tracing Interview Forms for Public Health Departments

BACKGROUND

In Wisconsin, COVID-19 case interview forms contain free text fields that need to be mined to identify potential outbreaks for targeted policy making. We developed an automated pipeline to ingest the free text into a pre-trained neural language model to identify businesses and facilities as outbreaks.

OBJECTIVE

We aimed to examine the precision and recall of our natural language processing pipeline against existing outbreaks and potentially new clusters.

METHODS

Data on cases of COVID-19 were extracted from the Wisconsin Electronic Disease Surveillance System (WEDSS) for Dane County between July 1, 2020, and June 30, 2021. Features from the case interview forms were fed into a Bidirectional Encoder Representations from Transformers (BERT) model that was fine-tuned for named entity recognition (NER). We also developed a novel location mapping tool to provide addresses for relevant NERs. Precision and recall were measured against manually verified outbreaks and valid addresses in WEDSS.

RESULTS

There were 46,798 cases of COVID-19 with 4,183,273 total BERT tokens and 15,051 unique tokens. The recall and precision of the NER tool were 0.67 (95 % CI 0.66-0.68) and 0.55 (95 % CI: 0.54-0.57), respectively. For the location mapping tool, the recall and precision were 0.93 (95% CI: 0.92-0.95) and 0.93 (95% CI: 0.92-0.95), respectively. Across monthly intervals, the NER tool identified more potential clusters than were verified in the WEDSS system.

CONCLUSIONS

We developed a novel pipeline of tools that identified existing outbreaks and novel clusters with associated addresses. Our pipeline ingests data from a statewide database and may be deployed to assist local health departments for targeted interventions.

Age and Injury Size Influence the Magnitude of Fecal Dysbiosis in Adult Burn Patients

Clinical studies have demonstrated that age 50 years or older is an independent risk factor associated with poor prognosis after burn injury, the second leading cause of traumatic injuries in the aged population. While mechanisms driving age-dependent postburn mortality are perplexing, changes in the intestinal microbiome, may contribute to the heightened, dysregulated systemic response seen in aging burn patients. The fecal microbiome from 22 patients admitted to a verified burn center from July 2018 to February 2019 was stratified based on the age of 50 years and total burn surface area (TBSA) size of ≥10%. Significant differences (P = .014) in overall microbiota community composition (ie, beta diversity) were measured across the four patient groups: young <10% TBSA, young ≥10% TBSA, older <10% TBSA, and older ≥10% TBSA. Differences in beta diversity were driven by %TBSA (P = .013) and trended with age (P = .087). Alpha diversity components, richness, evenness, and Shannon diversity were measured. We observed significant differences in bacterial species evenness (P = .0023) and Shannon diversity (P = .0033) between the groups. There were significant correlations between individual bacterial species and levels of short-chain fatty acids. Specifically, levels of fecal butyrate correlated with the presence of Enterobacteriaceae, an opportunistic gut pathogen, when elevated in burn patients lead to worsen outcomes. Overall, our findings reveal that age-specific changes in the fecal microbiome following burn injuries may contribute to immune system dysregulation in patients with varying TBSA burns and potentially lead to worsened clinical outcomes with heightened morbidity and mortality.

Image and structured data analysis for prognostication of health outcomes in patients presenting to the ED during the COVID-19 pandemic

BACKGROUND

Patients admitted to the emergency department (ED) with COVID-19 symptoms are routinely required to have chest radiographs and computed tomography (CT) scans. COVID-19 infection has been directly related to the development of acute respiratory distress syndrome (ARDS) and severe infections could lead to admission to intensive care and increased risk of death. The use of clinical data in machine learning models available at time of admission to ED can be used to assess possible risk of ARDS, the need for intensive care (admission to the Intensive Care Unit; ICU) as well as risk of mortality. In addition, chest radiographs can be inputted into a deep learning model to further assess these risks.

PURPOSE

This research aimed to develop machine and deep learning models using both structured clinical data and image data from the electronic health record (EHR) to predict adverse outcomes following ED admission.

MATERIALS AND METHODS

Light Gradient Boosting Machine (LightGBM) was used as the main machine learning algorithm using all clinical data including 42 variables. Compact models were also developed using the 15 most important variables to increase applicability of the models in clinical settings. To predict risk (or early stratified risk) of the aforementioned health outcome events, transfer learning from the CheXNet model was also implemented on the available data. This research utilized clinical data and chest radiographs of 3,571 patients, 18 years and older, admitted to the emergency department between 9th March 2020 and 29th October 2020 at Loyola University Medical Center.

MAIN FINDINGS

The research results show that we can detect COVID-19 infection (AUC = 0.790 (0.746-0.835)), predict the risk of developing ARDS (AUC = 0.781 (0.690-0.872), risk stratification of the need for ICU admission (AUC = 0.675 (0.620-0.713)) and mortality (AUC = 0.759 (0.678-0.840)) at moderate accuracy from both chest X-ray images and clinical data.

PRINCIPAL CONCLUSIONS

The results can help in clinical decision making, especially when addressing ARDS and mortality, during the assessment of patients admitted to the ED with or without COVID-19 symptoms.

Investigating Unhealthy Alcohol Use As an Independent Risk Factor for Increased COVID-19 Disease Severity: Observational Cross-sectional Study

BACKGROUND

Unhealthy alcohol use (UAU) is known to disrupt pulmonary immune mechanisms and increase the risk of acute respiratory distress syndrome in patients with pneumonia; however, little is known about the effects of UAU on outcomes in patients with COVID-19 pneumonia. To our knowledge, this is the first observational cross-sectional study that aims to understand the effect of UAU on the severity of COVID-19.

OBJECTIVE

We aim to determine if UAU is associated with more severe clinical presentation and worse health outcomes related to COVID-19 and if socioeconomic status, smoking, age, BMI, race/ethnicity, and pattern of alcohol use modify the risk.

METHODS

In this observational cross-sectional study that took place between January 1, 2020, and December 31, 2020, we ran a digital machine learning classifier on the electronic health record of patients who tested positive for SARS-CoV-2 via nasopharyngeal swab or had two COVID-19 International Classification of Disease, 10th Revision (ICD-10) codes to identify patients with UAU. After controlling for age, sex, ethnicity, BMI, smoking status, insurance status, and presence of ICD-10 codes for cancer, cardiovascular disease, and diabetes, we then performed a multivariable regression to examine the relationship between UAU and COVID-19 severity as measured by hospital care level (ie, emergency department admission, emergency department admission with ventilator, or death). We used a predefined cutoff with optimal sensitivity and specificity on the digital classifier to compare disease severity in patients with and without UAU. Models were adjusted for age, sex, race/ethnicity, BMI, smoking status, and insurance status.

RESULTS

Each incremental increase in the predicted probability from the digital alcohol classifier was associated with a greater odds risk for more severe COVID-19 disease (odds ratio 1.15, 95% CI 1.10-1.20). We found that patients in the unhealthy alcohol group had a greater odds risk to develop more severe disease (odds ratio 1.89, 95% CI 1.17-3.06), suggesting that UAU was associated with an 89% increase in the odds of being in a higher severity category.

CONCLUSIONS

In patients infected with SARS-CoV-2, UAU is an independent risk factor associated with greater disease severity and/or death.

Bias and fairness assessment of a natural language processing opioid misuse classifier: detection and mitigation of electronic health record data disadvantages across racial subgroups

OBJECTIVES

To assess fairness and bias of a previously validated machine learning opioid misuse classifier.

MATERIALS & METHODS

Two experiments were conducted with the classifier's original (n = 1000) and external validation (n = 53 974) datasets from 2 health systems. Bias was assessed via testing for differences in type II error rates across racial/ethnic subgroups (Black, Hispanic/Latinx, White, Other) using bootstrapped 95% confidence intervals. A local surrogate model was estimated to interpret the classifier's predictions by race and averaged globally from the datasets. Subgroup analyses and post-hoc recalibrations were conducted to attempt to mitigate biased metrics.

RESULTS

We identified bias in the false negative rate (FNR = 0.32) of the Black subgroup compared to the FNR (0.17) of the White subgroup. Top features included "heroin" and "substance abuse" across subgroups. Post-hoc recalibrations eliminated bias in FNR with minimal changes in other subgroup error metrics. The Black FNR subgroup had higher risk scores for readmission and mortality than the White FNR subgroup, and a higher mortality risk score than the Black true positive subgroup (P < .05).

DISCUSSION

The Black FNR subgroup had the greatest severity of disease and risk for poor outcomes. Similar features were present between subgroups for predicting opioid misuse, but inequities were present. Post-hoc mitigation techniques mitigated bias in type II error rate without creating substantial type I error rates. From model design through deployment, bias and data disadvantages should be systematically addressed.

CONCLUSION

Standardized, transparent bias assessments are needed to improve trustworthiness in clinical machine learning models.

Research Needs for Inpatient Management of Severe Alcohol Withdrawal Syndrome: An Official American Thoracic Society Research Statement

Background: Severe alcohol withdrawal syndrome (SAWS) is highly morbid, costly, and common among hospitalized patients, yet minimal evidence exists to guide inpatient management. Research needs in this field are broad, spanning the translational science spectrum. Goals: This research statement aims to describe what is known about SAWS, identify knowledge gaps, and offer recommendations for research in each domain of the Institute of Medicine T0-T4 continuum to advance the care of hospitalized patients who experience SAWS. Methods: Clinicians and researchers with unique and complementary expertise in basic, clinical, and implementation research related to unhealthy alcohol consumption and alcohol withdrawal were invited to participate in a workshop at the American Thoracic Society 2019 International Conference. The committee was subdivided into four groups on the basis of interest and expertise: T0-T1 (basic science research with translation to humans), T2 (research translating to patients), T3 (research translating to clinical practice), and T4 (research translating to communities). A medical librarian conducted a pragmatic literature search to facilitate this work, and committee members reviewed and supplemented the resulting evidence, identifying key knowledge gaps. Results: The committee identified several investigative opportunities to advance the care of patients with SAWS in each domain of the translational science spectrum. Major themes included 1) the need to investigate non-γ-aminobutyric acid pathways for alcohol withdrawal syndrome treatment; 2) harnessing retrospective and electronic health record data to identify risk factors and create objective severity scoring systems, particularly for acutely ill patients with SAWS; 3) the need for more robust comparative-effectiveness data to identify optimal SAWS treatment strategies; and 4) recommendations to accelerate implementation of effective treatments into practice. Conclusions: The dearth of evidence supporting management decisions for hospitalized patients with SAWS, many of whom require critical care, represents both a call to action and an opportunity for the American Thoracic Society and larger scientific communities to improve care for a vulnerable patient population. This report highlights basic, clinical, and implementation research that diverse experts agree will have the greatest impact on improving care for hospitalized patients with SAWS.

Identifying High-Risk Subphenotypes and Associated Harms From Delayed Antibiotic Orders and Delivery

OBJECTIVES

Early antibiotic administration is a central component of sepsis guidelines, and delays may increase mortality. However, prior studies have examined the delay to first antibiotic administration as a single time period even though it contains two distinct processes: antibiotic ordering and antibiotic delivery, which can each be targeted for improvement through different interventions. The objective of this study was to characterize and compare patients who experienced order or delivery delays, investigate the association of each delay type with mortality, and identify novel patient subphenotypes with elevated risk of harm from delays.

DESIGN

Retrospective analysis of multicenter inpatient data.

SETTING

Two tertiary care medical centers (2008-2018, 2006-2017) and four community-based hospitals (2008-2017).

PATIENTS

All patients admitted through the emergency department who met clinical criteria for infection.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Patient demographics, vitals, laboratory values, medication order and administration times, and in-hospital survival data were obtained from the electronic health record. Order and delivery delays were calculated for each admission. Adjusted logistic regression models were used to examine the relationship between each delay and in-hospital mortality. Causal forests, a machine learning method, was used to identify a high-risk subgroup. A total of 60,817 admissions were included, and delays occurred in 58% of patients. Each additional hour of order delay (odds ratio, 1.04; 95% CI, 1.03-1.05) and delivery delay (odds ratio, 1.05; 95% CI, 1.02-1.08) was associated with increased mortality. A patient subgroup identified by causal forests with higher comorbidity burden, greater organ dysfunction, and abnormal initial lactate measurements had a higher risk of death associated with delays (odds ratio, 1.07; 95% CI, 1.06-1.09 vs odds ratio, 1.02; 95% CI, 1.01-1.03).

CONCLUSIONS

Delays in antibiotic ordering and drug delivery are both associated with a similar increase in mortality. A distinct subgroup of high-risk patients exist who could be targeted for more timely therapy.