Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned

Background

Electronic health record (EHR)-based patient messages can contribute to burnout. Messages with a negative tone are particularly challenging to address. In this perspective, we describe our initial evaluation of large language model (LLM)-generated responses to negative EHR patient messages and contend that using LLMs to generate initial drafts may be feasible, although refinement will be needed.

Methods

A retrospective sample (n = 50) of negative patient messages was extracted from a health system EHR, de-identified, and inputted into an LLM (ChatGPT). Qualitative analyses were conducted to compare LLM responses to actual care team responses.

Results

Some LLM-generated draft responses varied from human responses in relational connection, informational content, and recommendations for next steps. Occasionally, the LLM draft responses could have potentially escalated emotionally charged conversations.

Conclusion

Further work is needed to optimize the use of LLMs for responding to negative patient messages in the EHR.

Sustainable Approach to Justice, Equity, Diversity, and Inclusion Through Better Quality Measurement

The US health care industry has broadly adopted performance and quality measures that are extracted from electronic health records and connected to payment incentives that hope to improve declining life expectancy and health status and reduce costs. While the development of a quality measurement infrastructure based on electronic health record data was an important first step in addressing US health outcomes, these metrics, reflecting the average performance across diverse populations, do not adequately adjust for population demographic differences, social determinants of health, or ecosystem vulnerability. Like society as a whole, health care must confront the powerful impact that social determinants of health, race, ethnicity, and other demographic variations have on key health care performance indicators and quality metrics. Tools that are currently available to capture and report the health status of Americans lack the granularity, complexity, and standardization needed to improve health and address disparities at the local level. In this article, we discuss the current and future state of electronic clinical quality measures through a lens of equity.

AI-Generated Draft Replies Integrated Into Health Records and Physicians' Electronic Communication

Importance

Timely tests are warranted to assess the association between generative artificial intelligence (GenAI) use and physicians' work efforts.

Objective

To investigate the association between GenAI-drafted replies for patient messages and physician time spent on answering messages and the length of replies.

Design, Setting, and Participants

Randomized waiting list quality improvement (QI) study from June to August 2023 in an academic health system. Primary care physicians were randomized to an immediate activation group and a delayed activation group. Data were analyzed from August to November 2023.

Exposure

Access to GenAI-drafted replies for patient messages.

Main Outcomes and Measures

Time spent (1) reading messages, (2) replying to messages, (3) length of replies, and (4) physician likelihood to recommend GenAI drafts. The a priori hypothesis was that GenAI drafts would be associated with less physician time spent reading and replying to messages. A mixed-effects model was used.

Results

Fifty-two physicians participated in this QI study, with 25 randomized to the immediate activation group and 27 randomized to the delayed activation group. A contemporary control group included 70 physicians. There were 18 female participants (72.0%) in the immediate group and 17 female participants (63.0%) in the delayed group; the median age range was 35-44 years in the immediate group and 45-54 years in the delayed group. The median (IQR) time spent reading messages in the immediate group was 26 (11-69) seconds at baseline, 31 (15-70) seconds 3 weeks after entry to the intervention, and 31 (14-70) seconds 6 weeks after entry. The delayed group's median (IQR) read time was 25 (10-67) seconds at baseline, 29 (11-77) seconds during the 3-week waiting period, and 32 (15-72) seconds 3 weeks after entry to the intervention. The contemporary control group's median (IQR) read times were 21 (9-54), 22 (9-63), and 23 (9-60) seconds in corresponding periods. The estimated association of GenAI was a 21.8% increase in read time (95% CI, 5.2% to 41.0%; P = .008), a -5.9% change in reply time (95% CI, -16.6% to 6.2%; P = .33), and a 17.9% increase in reply length (95% CI, 10.1% to 26.2%; P < .001). Participants recognized GenAI's value and suggested areas for improvement.

Conclusions and Relevance

In this QI study, GenAI-drafted replies were associated with significantly increased read time, no change in reply time, significantly increased reply length, and some perceived benefits. Rigorous empirical tests are necessary to further examine GenAI's performance. Future studies should examine patient experience and compare multiple GenAIs, including those with medical training.

Benefits and Challenges of Transitioning Occupational Health to an Enterprise Electronic Health Record

OBJECTIVE

Occupational health (OH) documentation has traditionally been separate from health system electronic health records (EHRs), but this can create patient safety and care continuity challenges. Herein, we describe outcomes and challenges of such integration including how one health system managed compliance with laws, regulations, and ethical principles concerning digital privacy.

METHODS

Occupational health integration with the enterprise EHR at the University of California San Diego Health was started in June 2021 and completed in December 2021.

RESULTS

Integrating with the enterprise EHR allowed for a secure telehealth system, faster visit times, digitization of questionnaires medical clearance forms, and improved reporting capabilities.

CONCLUSIONS

Integration and interoperability are fundamental building blocks to any OH EHR solution and will allow for evaluation of worker population trends, and targeted interventions to improve worker health status.

Development and implementation of an automated electronic health record-linked registry for emergency general surgery

INTRODUCTION

Despite adoption of the emergency general surgery (EGS) service by hospitals nationally, quality improvement (QI) and research for this patient population are challenging because of the lack of population-specific registries. Past efforts have been limited by difficulties in identifying EGS patients within institutions and labor-intensive approaches to data capture. Thus, we created an automated electronic health record (EHR)-linked registry for EGS.

METHODS

We built a registry within the Epic EHR at University of California San Diego for the EGS service. Existing EHR labels that identified patients seen by the EGS team were used to create our automated inclusion rules. Registry validation was performed using a retrospective cohort of EGS patients in a 30-month period and a 1-month prospective cohort. We created quality metrics that are updated and reported back to clinical teams in real time and obtained aggregate data to identify QI and research opportunities. A key metric tracked is clinic schedule rate, as we care that discontinuity postdischarge for the EGS population remains a challenge.

RESULTS

Our registry captured 1,992 patient encounters with 1,717 unique patients in the 30-month period. It had a false-positive EGS detection rate of 1.8%. In our 1-month prospective cohort, it had a false-positive EGS detection rate of 0% and sensitivity of 85%. For quality metrics analysis, we found that EGS patients who were seen as consults had significantly lower clinic schedule rates on discharge compared with those who were admitted to the EGS service (85% vs. 60.7%, p < 0.001).

CONCLUSION

An EHR-linked EGS registry can reliably conduct capture data automatically and support QI and research.

LEVEL OF EVIDENCE

Prognostic and epidemiological, level III.

Social Determinants of Health Data Availability for Patients with Eye Conditions

Purpose

To quantify and characterize social determinants of health (SDoH) data coverage using single-center electronic health records (EHRs) and the National Institutes of Health All of Us research program.

Design

Retrospective cohort study from June 2014 through June 2021.

Participants

Adults 18 years of age or older with a diagnosis of diabetic retinopathy, glaucoma, cataracts, or age-related macular degeneration.

Methods

For All of Us, research participants completed online survey forms as part of a nationwide prospective cohort study. In local EHRs, patients were selected based on diagnosis codes.

Main Outcome Measures

Social determinants of health data coverage, characterized by the proportion of each disease cohort with available data regarding demographics and socioeconomic factors.

Results

In All of Us, we identified 23 806 unique adult patients, of whom 2246 had a diagnosis of diabetic retinopathy, 13 448 had a diagnosis of glaucoma, 6634 had a diagnosis of cataracts, and 1478 had a diagnosis of age-related macular degeneration. Survey completion rates were high (99.5%-100%) across all cohorts for demographic information, overall health, income, education, and lifestyle. However, health care access (12.7%-29.4%), housing (0.7%-1.1%), social isolation (0.2%-0.3%), and food security (0-0.1%) showed significantly lower response rates. In local EHRs, we identified 80 548 adult patients, of whom 6616 had a diagnosis of diabetic retinopathy, 26 793 had a diagnosis of glaucoma, 40 427 had a diagnosis of cataracts, and 6712 had a diagnosis of age-related macular degeneration. High data coverage was found across all cohorts for variables related to tobacco use (82.84%-89.07%), alcohol use (77.45%-83.66%), and intravenous drug use (84.76%-93.14%). However, low data coverage (< 50% completion) was found for all other variables, including education, finances, social isolation, stress, physical activity, food insecurity, and transportation. We used chi-square testing to assess whether the data coverage varied across different disease cohorts and found that all fields varied significantly (P < 0.001).

Conclusions

The limited and highly variable data coverage in both local EHRs and All of Us highlights the need for researchers and providers to develop SDoH data collection strategies and to assemble complete datasets.

Relation of Statin Use Prior to Admission to Severity and Recovery Among COVID-19 Inpatients

The impact of statins, angiotensin-converting enzyme inhibitors and angiotensin II receptor blockers (ARBs) on coronavirus disease 2019 (COVID-19) severity and recovery is important given their high prevalence of use among individuals at risk for severe COVID-19. We studied the association between use of statin/angiotensin-converting enzyme inhibitors/ARB in the month before hospital admission, with risk of severe outcome, and with time to severe outcome or disease recovery, among patients hospitalized for COVID-19. We performed a retrospective single-center study of all patients hospitalized at University of California San Diego Health between February 10, 2020 and June 17, 2020 (n = 170 hospitalized for COVID-19, n = 5,281 COVID-negative controls). Logistic regression and competing risks analyses were used to investigate progression to severe disease (death or intensive care unit admission), and time to discharge without severe disease. Severe disease occurred in 53% of COVID-positive inpatients. Median time from hospitalization to severe disease was 2 days; median time to recovery was 7 days. Statin use prior to admission was associated with reduced risk of severe COVID-19 (adjusted OR 0.29, 95%CI 0.11 to 0.71, p < 0.01) and faster time to recovery among those without severe disease (adjusted HR for recovery 2.69, 95%CI 1.36 to 5.33, p < 0.01). The association between statin use and severe disease was smaller in the COVID-negative cohort (p for interaction = 0.07). There was potential evidence of faster time to recovery with ARB use (adjusted HR 1.92, 95%CI 0.81 to 4.56). In conclusion, statin use during the 30 days prior to admission for COVID-19 was associated with a lower risk of developing severe COVID-19, and a faster time to recovery among patients without severe disease.

COVID-19 TestNorm: A tool to normalize COVID-19 testing names to LOINC codes

Large observational data networks that leverage routine clinical practice data in electronic health records (EHRs) are critical resources for research on coronavirus disease 2019 (COVID-19). Data normalization is a key challenge for the secondary use of EHRs for COVID-19 research across institutions. In this study, we addressed the challenge of automating the normalization of COVID-19 diagnostic tests, which are critical data elements, but for which controlled terminology terms were published after clinical implementation. We developed a simple but effective rule-based tool called COVID-19 TestNorm to automatically normalize local COVID-19 testing names to standard LOINC (Logical Observation Identifiers Names and Codes) codes. COVID-19 TestNorm was developed and evaluated using 568 test names collected from 8 healthcare systems. Our results show that it could achieve an accuracy of 97.4% on an independent test set. COVID-19 TestNorm is available as an open-source package for developers and as an online Web application for end users (https://clamp.uth.edu/covid/loinc.php). We believe that it will be a useful tool to support secondary use of EHRs for research on COVID-19.

A Path to Clinical Quality Integration Through a Clinically Integrated Network: The Experience of an Academic Health System and Its Community Affiliates

INTRODUCTION

Academic medical centers (AMCs) and community physicians seeking to establish a clinically integrated network (CIN) may benefit from a road map to navigate the opportunities and challenges of such an organizational structure. Creating and participating in a CIN requires careful consideration, investment of time, financial resources, alignment of a new quality infrastructure, shared governance, and vision.

POTENTIAL BENEFITS, CHALLENGES, AND REGULATORY CONSIDERATIONS

Potential AMC benefits include geographic clinical expansion, the ability to provide care for a broader population of patients, a mechanism to collaborate with regional physician graduates, and an expansion of available teaching sites for trainees. Potential benefits to community practices include propagation of high-value care, enhanced access to evidence-based protocols and priority measures, preparation for value-based reimbursement structures, and connection to an institution that produces future health care practitioners. Challenges to CIN creation include goal alignment, trust between AMC and community partners, acceptance of common quality measures and benchmarks, access to shared data, and local adoption of quality improvement activities.

QUALITY AND INFORMATION TECHNOLOGY CONSIDERATIONS

At inception the mission was to create an innovative academic-community alliance delivering high-quality, high-value, personalized care. Defining the clinical quality goals, measurement, governance, and improvement strategy, as well as information technology structure and decision making, are described.

FUTURE DIRECTIONS

The network continues to grow and now includes more than 350 physicians, in 16 different specialties across 50 different independent medical practices throughout Southern California. We believe this builds a firm foundation for value-based health care.

Barriers to Implementing an Artificial Intelligence Model for Unplanned Readmissions

BACKGROUND

Electronic health record (EHR) vendors now offer "off-the-shelf" artificial intelligence (AI) models to client organizations. Our health system faced difficulties in promoting end-user utilization of a new AI model for predicting readmissions embedded in the EHR.

OBJECTIVES

The aim is to conduct a case study centered on identifying barriers to uptake/utilization.

METHODS

A qualitative study was conducted using interviews with stakeholders. The interviews were used to identify relevant stakeholders, understand current workflows, identify implementation barriers, and formulate future strategies.

RESULTS

We discovered substantial variation in existing workflows around readmissions. Some stakeholders did not perform any formal readmissions risk assessment. Others accustomed to using existing risk scores such as LACE+ had concerns about transitioning to a new model. Some stakeholders had existing workflows in place that could accommodate the new model, but they were not previously aware that the new model was in production. Concerns expressed by end-users included: whether the model's predictors were relevant to their work, need for adoption of additional workflow processes, need for training and change management, and potential for unintended consequences (e.g., increased health care resource utilization due to potentially over-referring discharged patients to home health services).

CONCLUSION

AI models for risk stratification, even if "off-the-shelf" by design, are unlikely to be "plug-and-play" in health care settings. Seeking out key stakeholders and defining clear use cases early in the implementation process can better facilitate utilization of these models.

A case study of the 1115 waiver using population health informatics to address disparities

As participants in the California Medicaid 1115 waiver, the University of California San Diego Health (UCSDH) used population health informatics tools to address health disparities. This case study describes a modern application of health informatics to improve data capture, describe health disparities through demographic stratification, and drive reliable care through electronic medical record-based registries. We provide a details in our successful approach using (1) standardized collection of race, ethnicity, language, sexual orientation, and gender identity data, (2) stratification of 8 quality measures by demographic profile, and (3) improved quality performance through registries for wellness, social determinants of health, and chronic disease. A strong population health platform paired with executive support, physician leadership, education and training, and workflow redesign can improve the representation of diversity and drive reliable processes for care delivery that improve health equity.

Utility of clinical assessment, imaging, and cryptococcal antigen titer to predict AIDS-related complicated forms of cryptococcal meningitis

BACKGROUND

This study aimed to evaluate the prevalence and predictors of AIDS-related complicated cryptococcal meningitis. The outcome was complicated cryptococcal meningitis: prolonged (>/= 14 days) altered mental status, persistent (>/= 14 days) focal neurologic findings, cerebrospinal fluid (CSF) shunt placement or death. Predictor variable operating characteristics were estimated using receiver operating characteristic curve (ROC) analysis. Multivariate analysis identified independent predictors of the outcome.

RESULTS

From 1990-2009, 82 patients with first episode of cryptococcal meningitis were identified. Of these, 14 (17%) met criteria for complicated forms of cryptococcal meningitis (prolonged altered mental status 6, persistent focal neurologic findings 7, CSF surgical shunt placement 8, and death 5). Patients with complicated cryptococcal meningitis had higher frequency of baseline focal neurological findings, head computed tomography (CT) abnormalities, mean CSF opening pressure, and cryptococcal antigen (CRAG) titers in serum and CSF. ROC area of log2 serum and CSF CRAG titers to predict complicated forms of cryptococcal meningitis were comparable, 0.78 (95%CI: 0.66 to 0.90) vs. 0.78 (95% CI: 0.67 to 0.89), respectively (chi2, p = 0.95). The ROC areas to predict the outcomes were similar for CSF pressure and CSF CRAG titers. In a multiple logistic regression model, the following were significant predictors of the outcome: baseline focal neurologic findings, head CT abnormalities and log2 CSF CRAG titer.

CONCLUSIONS

During initial clinical evaluation, a focal neurologic exam, abnormal head CT and large cryptococcal burden measured by CRAG titer are associated with the outcome of complicated cryptococcal meningitis following 2 weeks from antifungal therapy initiation.

Treatment of HIV-related inflammatory cerebral cryptococcoma with adalimumab

Cryptococcomas have been described in AIDS patients in the setting of immune reconstitution inflammatory syndrome. We report the first case of human immunodeficiency virus-related inflammatory cerebral cryptococcoma to be treated with a recombinant human monoclonal tumor necrosis factor antagonist.