Development and evaluation of an interoperable natural language processing system for identifying pneumonia across clinical settings of care and institutions

Diagnostic Discordance, Uncertainty, and Treatment Ambiguity in Community-Acquired Pneumonia : A National Cohort Study of 115 U.S. Veterans Affairs Hospitals

BACKGROUND

Evidence-based practice in community-acquired pneumonia often assumes an accurate initial diagnosis.

OBJECTIVE

To examine the evolution of pneumonia diagnoses among patients hospitalized from the emergency department (ED).

DESIGN

Retrospective nationwide cohort.

SETTING

118 U.S. Veterans Affairs medical centers.

PATIENTS

Aged 18 years or older and hospitalized from the ED between 1 January 2015 and 31 January 2022.

MEASUREMENTS

Discordances between initial pneumonia diagnosis, discharge diagnosis, and radiographic diagnosis identified by natural language processing of clinician text, diagnostic coding, and antimicrobial treatment. Expressions of uncertainty in clinical notes, patient illness severity, treatments, and outcomes were compared.

RESULTS

Among 2 383 899 hospitalizations, 13.3% received an initial or discharge diagnosis and treatment of pneumonia: 9.1% received an initial diagnosis and 10.0% received a discharge diagnosis. Discordances between initial and discharge occurred in 57%. Among patients discharged with a pneumonia diagnosis and positive initial chest image, 33% lacked an initial diagnosis. Among patients diagnosed initially, 36% lacked a discharge diagnosis and 21% lacked positive initial chest imaging. Uncertainty was frequently expressed in clinical notes (58% in ED; 48% at discharge); 27% received diuretics, 36% received corticosteroids, and 10% received antibiotics, corticosteroids, and diuretics within 24 hours. Patients with discordant diagnoses had greater uncertainty and received more additional treatments, but only patients lacking an initial pneumonia diagnosis had higher 30-day mortality than concordant patients (14.4% [95% CI, 14.1% to 14.7%] vs. 10.6% [CI, 10.4% to 10.7%]). Patients with diagnostic discordance were more likely to present to high-complexity facilities with high ED patient load and inpatient census.

LIMITATION

Retrospective analysis; did not examine causal relationships.

CONCLUSION

More than half of all patients hospitalized and treated for pneumonia had discordant diagnoses from initial presentation to discharge. Treatments for other diagnoses and expressions of uncertainty were common. These findings highlight the need to recognize diagnostic uncertainty and treatment ambiguity in research and practice of pneumonia-related care.

PRIMARY FUNDING SOURCE

The Gordon and Betty Moore Foundation.

Pneumonia diagnosis performance in the emergency department: a mixed-methods study about clinicians' experiences and exploration of individual differences and response to diagnostic performance feedback

OBJECTIVES

We sought to (1) characterize the process of diagnosing pneumonia in an emergency department (ED) and (2) examine clinician reactions to a clinician-facing diagnostic discordance feedback tool.

MATERIALS AND METHODS

We designed a diagnostic feedback tool, using electronic health record data from ED clinicians' patients to establish concordance or discordance between ED diagnosis, radiology reports, and hospital discharge diagnosis for pneumonia. We conducted semistructured interviews with 11 ED clinicians about pneumonia diagnosis and reactions to the feedback tool. We administered surveys measuring individual differences in mindset beliefs, comfort with feedback, and feedback tool usability. We qualitatively analyzed interview transcripts and descriptively analyzed survey data.

RESULTS

Thematic results revealed: (1) the diagnostic process for pneumonia in the ED is characterized by diagnostic uncertainty and may be secondary to goals to treat and dispose the patient; (2) clinician diagnostic self-evaluation is a fragmented, inconsistent process of case review and follow-up that a feedback tool could fill; (3) the feedback tool was described favorably, with task and normative feedback harnessing clinician values of high-quality patient care and personal excellence; and (4) strong reactions to diagnostic feedback varied from implicit trust to profound skepticism about the validity of the concordance metric. Survey results suggested a relationship between clinicians' individual differences in learning and failure beliefs, feedback experience, and usability ratings.

DISCUSSION AND CONCLUSION

Clinicians value feedback on pneumonia diagnoses. Our results highlight the importance of feedback about diagnostic performance and suggest directions for considering individual differences in feedback tool design and implementation.

Assessing Diagnostic Performance

Assessing Diagnostic PerformanceDiagnosis is an action and a goal in medicine. This article is the introduction to a series of review articles on varying facets of diagnosis. Clinical diagnosis is an exercise in classification; that is, placing the patient's condition in the correct diagnostic category. However, consideration must also be given to the performance objective, whether it is technical performance of a test, acquiring diagnostic information with respect to clinical management for an individual or a population's health outcomes, or cost-effectiveness and equity of care.

An Algorithm to Assess Guideline Concordance of Antibiotic Choice in Community-Acquired Pneumonia

BACKGROUND AND OBJECTIVE

This study aimed to develop and evaluate an algorithm to reduce the chart review burden of improvement efforts by automatically labeling antibiotic selection as either guideline-concordant or -discordant based on electronic health record data for patients with community-acquired pneumonia (CAP).

METHODS

We developed a 3-part algorithm using structured and unstructured data to assess adherence to an institutional CAP clinical practice guideline. The algorithm was applied to retrospective data for patients seen with CAP from 2017 to 2019 at a tertiary children's hospital. Performance metrics included positive predictive value (precision), sensitivity (recall), and F1 score (harmonized mean), with macro-weighted averages. Two physician reviewers independently assigned "actual" labels based on manual chart review.

RESULTS

Of 1345 patients with CAP, 893 were included in the training cohort and 452 in the validation cohort. Overall, the model correctly labeled 435 of 452 (96%) patients. Of the 286 patients who met guideline inclusion criteria, 193 (68%) were labeled as having received guideline-concordant antibiotics, 48 (17%) were labeled as likely in a scenario in which deviation from the clinical practice guideline was appropriate, and 45 (16%) were given the final label of "possibly discordant, needs review." The sensitivity was 0.96, the positive predictive value was 0.97, and the F1 was 0.96.

CONCLUSIONS

An automated algorithm that uses structured and unstructured electronic health record data can accurately assess the guideline concordance of antibiotic selection for CAP. This tool has the potential to improve the efficiency of improvement efforts by reducing the manual chart review needed for quality measurement.

Using natural language processing to study homelessness longitudinally with electronic health record data subject to irregular observations

The Electronic Health Record (EHR) contains information about social determinants of health (SDoH) such as homelessness. Much of this information is contained in clinical notes and can be extracted using natural language processing (NLP). This data can provide valuable information for researchers and policymakers studying long-term housing outcomes for individuals with a history of homelessness. However, studying homelessness longitudinally in the EHR is challenging due to irregular observation times. In this work, we applied an NLP system to extract housing status for a cohort of patients in the US Department of Veterans Affairs (VA) over a three-year period. We then applied inverse intensity weighting to adjust for the irregularity of observations, which was used generalized estimating equations to estimate the probability of unstable housing each day after entering a VA housing assistance program. Our methods generate unique insights into the long-term outcomes of individuals with a history of homelessness and demonstrate the potential for using EHR data for research and policymaking.

Assessing longitudinal housing status using Electronic Health Record data: a comparison of natural language processing, structured data, and patient-reported history

Introduction

Measuring long-term housing outcomes is important for evaluating the impacts of services for individuals with homeless experience. However, assessing long-term housing status using traditional methods is challenging. The Veterans Affairs (VA) Electronic Health Record (EHR) provides detailed data for a large population of patients with homeless experiences and contains several indicators of housing instability, including structured data elements (e.g., diagnosis codes) and free-text clinical narratives. However, the validity of each of these data elements for measuring housing stability over time is not well-studied.

Methods

We compared VA EHR indicators of housing instability, including information extracted from clinical notes using natural language processing (NLP), with patient-reported housing outcomes in a cohort of homeless-experienced Veterans.

Results

NLP achieved higher sensitivity and specificity than standard diagnosis codes for detecting episodes of unstable housing. Other structured data elements in the VA EHR showed promising performance, particularly when combined with NLP.

Discussion

Evaluation efforts and research studies assessing longitudinal housing outcomes should incorporate multiple data sources of documentation to achieve optimal performance.