1
|
Jones BE, Chapman AB, Ying J, Rutter ED, Nevers MR, Baker A, Dean NC, Fix ML, Singh H, Cosby KS, Taber PA, Weir CD, Jones MM, Samore MH, Butler JM. Diagnostic Discordance, Uncertainty, and Treatment Ambiguity in Community-Acquired Pneumonia : A National Cohort Study of 115 U.S. Veterans Affairs Hospitals. Ann Intern Med 2024. [PMID: 39102729 DOI: 10.7326/m23-2505] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 08/07/2024] Open
Abstract
BACKGROUND Evidence-based practice in community-acquired pneumonia often assumes an accurate initial diagnosis. OBJECTIVE To examine the evolution of pneumonia diagnoses among patients hospitalized from the emergency department (ED). DESIGN Retrospective nationwide cohort. SETTING 118 U.S. Veterans Affairs medical centers. PATIENTS Aged 18 years or older and hospitalized from the ED between 1 January 2015 and 31 January 2022. MEASUREMENTS Discordances between initial pneumonia diagnosis, discharge diagnosis, and radiographic diagnosis identified by natural language processing of clinician text, diagnostic coding, and antimicrobial treatment. Expressions of uncertainty in clinical notes, patient illness severity, treatments, and outcomes were compared. RESULTS Among 2 383 899 hospitalizations, 13.3% received an initial or discharge diagnosis and treatment of pneumonia: 9.1% received an initial diagnosis and 10.0% received a discharge diagnosis. Discordances between initial and discharge occurred in 57%. Among patients discharged with a pneumonia diagnosis and positive initial chest image, 33% lacked an initial diagnosis. Among patients diagnosed initially, 36% lacked a discharge diagnosis and 21% lacked positive initial chest imaging. Uncertainty was frequently expressed in clinical notes (58% in ED; 48% at discharge); 27% received diuretics, 36% received corticosteroids, and 10% received antibiotics, corticosteroids, and diuretics within 24 hours. Patients with discordant diagnoses had greater uncertainty and received more additional treatments, but only patients lacking an initial pneumonia diagnosis had higher 30-day mortality than concordant patients (14.4% [95% CI, 14.1% to 14.7%] vs. 10.6% [CI, 10.4% to 10.7%]). Patients with diagnostic discordance were more likely to present to high-complexity facilities with high ED patient load and inpatient census. LIMITATION Retrospective analysis; did not examine causal relationships. CONCLUSION More than half of all patients hospitalized and treated for pneumonia had discordant diagnoses from initial presentation to discharge. Treatments for other diagnoses and expressions of uncertainty were common. These findings highlight the need to recognize diagnostic uncertainty and treatment ambiguity in research and practice of pneumonia-related care. PRIMARY FUNDING SOURCE The Gordon and Betty Moore Foundation.
Collapse
Affiliation(s)
- Barbara E Jones
- Division of Pulmonary & Critical Care Medicine, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (B.E.J.)
| | - Alec B Chapman
- Division of Epidemiology, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (A.B.C., J.Y., M.R.N., M.M.J., M.H.S.)
| | - Jian Ying
- Division of Epidemiology, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (A.B.C., J.Y., M.R.N., M.M.J., M.H.S.)
| | - Elizabeth D Rutter
- Department of Emergency Medicine, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (E.D.R., A.B.)
| | - McKenna R Nevers
- Division of Epidemiology, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (A.B.C., J.Y., M.R.N., M.M.J., M.H.S.)
| | - Alden Baker
- Department of Emergency Medicine, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (E.D.R., A.B.)
| | - Nathan C Dean
- Division of Pulmonary and Critical Care Medicine, Intermountain Health and University of Utah, Murray, Utah (N.C.D.)
| | - Megan L Fix
- Department of Emergency Medicine, University of Utah Healthcare System, Salt Lake City, Utah (M.L.F.)
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas (H.S.)
| | - Karen S Cosby
- Department of Emergency Medicine, Cook County Hospital, Rush Medical College, Chicago, Illinois (K.S.C.)
| | - Peter A Taber
- Department of Biomedical Informatics, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (P.A.T., C.D.W.)
| | - Charlene D Weir
- Department of Biomedical Informatics, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (P.A.T., C.D.W.)
| | - Makoto M Jones
- Division of Epidemiology, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (A.B.C., J.Y., M.R.N., M.M.J., M.H.S.)
| | - Matthew H Samore
- Division of Epidemiology, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (A.B.C., J.Y., M.R.N., M.M.J., M.H.S.)
| | - Jorie M Butler
- Department of Biomedical Informatics, and Division of Geriatrics, Department of Internal Medicine, University of Utah and Salt Lake City VA Healthcare System, Salt Lake City, Utah (J.M.B.)
| |
Collapse
|
2
|
Butler JM, Taft T, Taber P, Rutter E, Fix M, Baker A, Weir C, Nevers M, Classen D, Cosby K, Jones M, Chapman A, Jones BE. Pneumonia diagnosis performance in the emergency department: a mixed-methods study about clinicians' experiences and exploration of individual differences and response to diagnostic performance feedback. J Am Med Inform Assoc 2024; 31:1503-1513. [PMID: 38796835 PMCID: PMC11187426 DOI: 10.1093/jamia/ocae112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 03/25/2024] [Accepted: 05/22/2024] [Indexed: 05/29/2024] Open
Abstract
OBJECTIVES We sought to (1) characterize the process of diagnosing pneumonia in an emergency department (ED) and (2) examine clinician reactions to a clinician-facing diagnostic discordance feedback tool. MATERIALS AND METHODS We designed a diagnostic feedback tool, using electronic health record data from ED clinicians' patients to establish concordance or discordance between ED diagnosis, radiology reports, and hospital discharge diagnosis for pneumonia. We conducted semistructured interviews with 11 ED clinicians about pneumonia diagnosis and reactions to the feedback tool. We administered surveys measuring individual differences in mindset beliefs, comfort with feedback, and feedback tool usability. We qualitatively analyzed interview transcripts and descriptively analyzed survey data. RESULTS Thematic results revealed: (1) the diagnostic process for pneumonia in the ED is characterized by diagnostic uncertainty and may be secondary to goals to treat and dispose the patient; (2) clinician diagnostic self-evaluation is a fragmented, inconsistent process of case review and follow-up that a feedback tool could fill; (3) the feedback tool was described favorably, with task and normative feedback harnessing clinician values of high-quality patient care and personal excellence; and (4) strong reactions to diagnostic feedback varied from implicit trust to profound skepticism about the validity of the concordance metric. Survey results suggested a relationship between clinicians' individual differences in learning and failure beliefs, feedback experience, and usability ratings. DISCUSSION AND CONCLUSION Clinicians value feedback on pneumonia diagnoses. Our results highlight the importance of feedback about diagnostic performance and suggest directions for considering individual differences in feedback tool design and implementation.
Collapse
Affiliation(s)
- Jorie M Butler
- Department of Biomedical Informatics, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
- Department of Internal Medicine, Division of Geriatrics, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84132, United States
- Salt Lake City VA Informatics Decision-Enhancement and Analytic Sciences (IDEAS) Center for Innovation, Salt Lake City, UT 84148, United States
- Geriatrics Research, Education, and Clinical Center (GRECC), VA Salt Lake City Health Care System, Salt Lake City, UT 84148, United States
| | - Teresa Taft
- Department of Biomedical Informatics, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Peter Taber
- Department of Biomedical Informatics, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
- Salt Lake City VA Informatics Decision-Enhancement and Analytic Sciences (IDEAS) Center for Innovation, Salt Lake City, UT 84148, United States
| | - Elizabeth Rutter
- Department of Emergency Medicine, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Megan Fix
- Department of Emergency Medicine, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Alden Baker
- Department of Family and Preventive Medicine, Division of Physician Assistant Studies, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Charlene Weir
- Department of Biomedical Informatics, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - McKenna Nevers
- Department of Internal Medicine, Division of Epidemiology, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - David Classen
- Department of Internal Medicine, Division of Epidemiology, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Karen Cosby
- Department of Emergency Medicine, Cook County Hospital, Rush Medical College, Chicago, IL 60612, United States
| | - Makoto Jones
- Salt Lake City VA Informatics Decision-Enhancement and Analytic Sciences (IDEAS) Center for Innovation, Salt Lake City, UT 84148, United States
- Department of Internal Medicine, Division of Epidemiology, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Alec Chapman
- Department of Population Health Sciences, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| | - Barbara E Jones
- Salt Lake City VA Informatics Decision-Enhancement and Analytic Sciences (IDEAS) Center for Innovation, Salt Lake City, UT 84148, United States
- Department of Internal Medicine, Division of Pulmonology, University of Utah Spencer Fox Eccles School of Medicine, Salt Lake City, UT 84108, United States
| |
Collapse
|
3
|
Cosby K, Yang D, Fineberg HV. Assessing Diagnostic Performance. NEJM EVIDENCE 2024; 3:EVIDra2300232. [PMID: 38320492 DOI: 10.1056/evidra2300232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/08/2024]
Abstract
Assessing Diagnostic PerformanceDiagnosis is an action and a goal in medicine. This article is the introduction to a series of review articles on varying facets of diagnosis. Clinical diagnosis is an exercise in classification; that is, placing the patient's condition in the correct diagnostic category. However, consideration must also be given to the performance objective, whether it is technical performance of a test, acquiring diagnostic information with respect to clinical management for an individual or a population's health outcomes, or cost-effectiveness and equity of care.
Collapse
Affiliation(s)
- Karen Cosby
- Gordon and Betty Moore Foundation, Palo Alto, CA
| | | | | |
Collapse
|
4
|
Yarahuan JK, Kisvarday S, Kim E, Yan AP, Nakamura MM, Jones SB, Hron JD. An Algorithm to Assess Guideline Concordance of Antibiotic Choice in Community-Acquired Pneumonia. Hosp Pediatr 2024; 14:137-145. [PMID: 38287897 PMCID: PMC10823186 DOI: 10.1542/hpeds.2023-007418] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/31/2024]
Abstract
BACKGROUND AND OBJECTIVE This study aimed to develop and evaluate an algorithm to reduce the chart review burden of improvement efforts by automatically labeling antibiotic selection as either guideline-concordant or -discordant based on electronic health record data for patients with community-acquired pneumonia (CAP). METHODS We developed a 3-part algorithm using structured and unstructured data to assess adherence to an institutional CAP clinical practice guideline. The algorithm was applied to retrospective data for patients seen with CAP from 2017 to 2019 at a tertiary children's hospital. Performance metrics included positive predictive value (precision), sensitivity (recall), and F1 score (harmonized mean), with macro-weighted averages. Two physician reviewers independently assigned "actual" labels based on manual chart review. RESULTS Of 1345 patients with CAP, 893 were included in the training cohort and 452 in the validation cohort. Overall, the model correctly labeled 435 of 452 (96%) patients. Of the 286 patients who met guideline inclusion criteria, 193 (68%) were labeled as having received guideline-concordant antibiotics, 48 (17%) were labeled as likely in a scenario in which deviation from the clinical practice guideline was appropriate, and 45 (16%) were given the final label of "possibly discordant, needs review." The sensitivity was 0.96, the positive predictive value was 0.97, and the F1 was 0.96. CONCLUSIONS An automated algorithm that uses structured and unstructured electronic health record data can accurately assess the guideline concordance of antibiotic selection for CAP. This tool has the potential to improve the efficiency of improvement efforts by reducing the manual chart review needed for quality measurement.
Collapse
Affiliation(s)
- Julia K.W. Yarahuan
- />Department of Pediatrics, Divisions of General Pediatrics
- Department of Pediatrics, Emory University School of Medicine and Division of Hospital Medicine, Children’s Healthcare of Atlanta, Atlanta, Georgia
| | | | | | - Adam P. Yan
- Hematology and Oncology
- Department of Pediatrics, The University of Toronto and Division of Hematology Oncology, The Hospital of Sick Children, Toronto, Ontario, Canada
| | | | - Sarah B. Jones
- Antimicrobial Stewardship Program
- Department of Pharmacy, Boston Children’s Hospital, Boston, Massachusetts
| | | |
Collapse
|
5
|
Chapman AB, Scharfstein DO, Montgomery AE, Byrne T, Suo Y, Effiong A, Velasquez T, Pettey W, Nelson RE. Using natural language processing to study homelessness longitudinally with electronic health record data subject to irregular observations. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2024; 2023:894-903. [PMID: 38222404 PMCID: PMC10785905] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
The Electronic Health Record (EHR) contains information about social determinants of health (SDoH) such as homelessness. Much of this information is contained in clinical notes and can be extracted using natural language processing (NLP). This data can provide valuable information for researchers and policymakers studying long-term housing outcomes for individuals with a history of homelessness. However, studying homelessness longitudinally in the EHR is challenging due to irregular observation times. In this work, we applied an NLP system to extract housing status for a cohort of patients in the US Department of Veterans Affairs (VA) over a three-year period. We then applied inverse intensity weighting to adjust for the irregularity of observations, which was used generalized estimating equations to estimate the probability of unstable housing each day after entering a VA housing assistance program. Our methods generate unique insights into the long-term outcomes of individuals with a history of homelessness and demonstrate the potential for using EHR data for research and policymaking.
Collapse
Affiliation(s)
- Alec B Chapman
- IDEAS Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, UT
- Department of Population Health Sciences, University of Utah School of Medicine, Salt Lake City, UT
| | - Daniel O Scharfstein
- Department of Population Health Sciences, University of Utah School of Medicine, Salt Lake City, UT
| | | | - Thomas Byrne
- National Center on Homelessness among Veterans
- School of Social Work, Boston University, Boston, MA
- Center for Healthcare Organization and Implementation Research, Bedford VA Medical Center, Bedford, MA
| | - Ying Suo
- IDEAS Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, UT
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT
| | - Atim Effiong
- IDEAS Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, UT
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT
| | - Tania Velasquez
- IDEAS Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, UT
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT
| | - Warren Pettey
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT
| | - Richard E Nelson
- IDEAS Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, UT
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT
- National Center on Homelessness among Veterans
| |
Collapse
|
6
|
Chapman AB, Cordasco K, Chassman S, Panadero T, Agans D, Jackson N, Clair K, Nelson R, Montgomery AE, Tsai J, Finley E, Gabrielian S. Assessing longitudinal housing status using Electronic Health Record data: a comparison of natural language processing, structured data, and patient-reported history. Front Artif Intell 2023; 6:1187501. [PMID: 37293237 PMCID: PMC10244644 DOI: 10.3389/frai.2023.1187501] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Accepted: 05/05/2023] [Indexed: 06/10/2023] Open
Abstract
Introduction Measuring long-term housing outcomes is important for evaluating the impacts of services for individuals with homeless experience. However, assessing long-term housing status using traditional methods is challenging. The Veterans Affairs (VA) Electronic Health Record (EHR) provides detailed data for a large population of patients with homeless experiences and contains several indicators of housing instability, including structured data elements (e.g., diagnosis codes) and free-text clinical narratives. However, the validity of each of these data elements for measuring housing stability over time is not well-studied. Methods We compared VA EHR indicators of housing instability, including information extracted from clinical notes using natural language processing (NLP), with patient-reported housing outcomes in a cohort of homeless-experienced Veterans. Results NLP achieved higher sensitivity and specificity than standard diagnosis codes for detecting episodes of unstable housing. Other structured data elements in the VA EHR showed promising performance, particularly when combined with NLP. Discussion Evaluation efforts and research studies assessing longitudinal housing outcomes should incorporate multiple data sources of documentation to achieve optimal performance.
Collapse
Affiliation(s)
- Alec B. Chapman
- Informatics, Decision-Enhancement and Analytic Sciences (IDEAS) Center, Salt Lake City Veterans Affairs Healthcare System, Salt Lake City, UT, United States
- Division of Epidemiology, University of Utah, School of Medicine, Salt Lake City, UT, United States
| | - Kristina Cordasco
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States
| | - Stephanie Chassman
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
- Desert Pacific Mental Illness Research, Education, and Clinical Center (MIRECC), Veterans Affairs Greater Los Angeles, Los Angeles, CA, United States
| | - Talia Panadero
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
- Department of Epidemiology, Fielding School of Public Health, University of California, Los Angeles, Los Angeles, CA, United States
| | - Dylan Agans
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
- Department of Community Health Sciences, Fielding School of Public Health, University of California, Los Angeles, Los Angeles, CA, United States
| | - Nicholas Jackson
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
- Department of Medicine Statistics Core, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States
| | - Kimberly Clair
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
| | - Richard Nelson
- Informatics, Decision-Enhancement and Analytic Sciences (IDEAS) Center, Salt Lake City Veterans Affairs Healthcare System, Salt Lake City, UT, United States
- Division of Epidemiology, University of Utah, School of Medicine, Salt Lake City, UT, United States
| | - Ann Elizabeth Montgomery
- United States Department of Veteran Affairs, Birmingham Veterans Affairs Health Care System, Birmingham, AL, United States
- School of Public Health, University of Alabama at Birmingham, Birmingham, AL, United States
| | - Jack Tsai
- National Homeless Programs Office, United States Department of Veterans Affairs, Washington, DC, United States
| | - Erin Finley
- United States Department of Veteran Affairs, Birmingham Veterans Affairs Health Care System, Birmingham, AL, United States
| | - Sonya Gabrielian
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, CA, United States
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States
| |
Collapse
|