Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use

Using large language model to guide patients to create efficient and comprehensive clinical care message

OBJECTIVE

This study aims to investigate the feasibility of using Large Language Models (LLMs) to engage with patients at the time they are drafting a question to their healthcare providers, and generate pertinent follow-up questions that the patient can answer before sending their message, with the goal of ensuring that their healthcare provider receives all the information they need to safely and accurately answer the patient's question, eliminating back-and-forth messaging, and the associated delays and frustrations.

METHODS

We collected a dataset of patient messages sent between January 1, 2022 to March 7, 2023 at Vanderbilt University Medical Center. Two internal medicine physicians identified 7 common scenarios. We used 3 LLMs to generate follow-up questions: (1) Comprehensive LLM Artificial Intelligence Responder (CLAIR): a locally fine-tuned LLM, (2) GPT4 with a simple prompt, and (3) GPT4 with a complex prompt. Five physicians rated them with the actual follow-ups written by healthcare providers on clarity, completeness, conciseness, and utility.

RESULTS

For five scenarios, our CLAIR model had the best performance. The GPT4 model received higher scores for utility and completeness but lower scores for clarity and conciseness. CLAIR generated follow-up questions with similar clarity and conciseness as the actual follow-ups written by healthcare providers, with higher utility than healthcare providers and GPT4, and lower completeness than GPT4, but better than healthcare providers.

CONCLUSION

LLMs can generate follow-up patient messages designed to clarify a medical question that compares favorably to those generated by healthcare providers.

The relationship between telemedicine tools and physician satisfaction, quality of care, and patient visits during the COVID-19 pandemic

OBJECTIVE

The objective of our study is to investigate the impacts of telemedicine technology and its specific tools on physicians' overall satisfaction, quality of care, and percentage of patient visits in ambulatory care settings after the COVID-19 lockdowns.

MATERIALS AND METHODS

Data for our analysis was sourced from the 2021 annual National Electronic Health Records Survey (NEHRS), which included 1,875 complete questionnaire responses from physicians in the 2021 NEHRS. We used regression models to test the effects of telemedicine on physicians' overall satisfaction, quality of care, and percentage of patients' visits.

RESULTS

We report that telemedicine technology has significant positive effects on physicians' satisfaction with telemedicine and quality of care evaluation, both at an aggregate level and at the disaggregate levels of individual telemedicine features, and partially significant effects on patients' telemedicine visits.

DISCUSSION

Telemedicine features that contributed significantly to physician satisfaction and quality of care evaluation were telephone, videoconferencing, standalone telemedicine platform, and telemedicine platform integrated with EHR, while only telephone and integrated telemedicine platform contributed significantly to patients' telemedicine visits.

CONCLUSION

For telemedicine research and practice, this study confirms that telemedicine improves physician satisfaction and quality of care perceptions and will therefore be preferred by physicians. However, telemedicine has a mixed impact on percentage of patient visits, which suggests that providers may have to work harder to regularize telemedicine acceptance among patients in the post-COVID era.

Harnessing artificial intelligence in bariatric surgery: comparative analysis of ChatGPT-4, Bing, and Bard in generating clinician-level bariatric surgery recommendations

BACKGROUND

The formulation of clinical recommendations pertaining to bariatric surgery is essential in guiding healthcare professionals. However, the extensive and continuously evolving body of literature in bariatric surgery presents considerable challenge for staying abreast of latest developments and efficient information acquisition. Artificial intelligence (AI) has the potential to streamline access to the salient points of clinical recommendations in bariatric surgery.

OBJECTIVES

The study aims to appraise the quality and readability of AI-chat-generated answers to frequently asked clinical inquiries in the field of bariatric and metabolic surgery.

SETTING

Remote.

METHODS

Question prompts inputted into AI large language models (LLMs) and were created based on pre-existing clinical practice guidelines regarding bariatric and metabolic surgery. The prompts were queried into 3 LLMs: OpenAI ChatGPT-4, Microsoft Bing, and Google Bard. The responses from each LLM were entered into a spreadsheet for randomized and blinded duplicate review. Accredited bariatric surgeons in North America independently assessed appropriateness of each recommendation using a 5-point Likert scale. Scores of 4 and 5 were deemed appropriate, while scores of 1-3 indicated lack of appropriateness. A Flesch Reading Ease (FRE) score was calculated to assess the readability of responses generated by each LLMs.

RESULTS

There was a significant difference between the 3 LLMs in their 5-point Likert scores, with mean values of 4.46 (SD .82), 3.89 (.80), and 3.11 (.72) for ChatGPT-4, Bard, and Bing (P < .001). There was a significant difference between the 3 LLMs in the proportion of appropriate answers, with ChatGPT-4 at 85.7%, Bard at 74.3%, and Bing at 25.7% (P < .001). The mean FRE scores for ChatGPT-4, Bard, and Bing, were 21.68 (SD 2.78), 42.89 (4.03), and 14.64 (5.09), respectively, with higher scores representing easier readability.

CONCLUSIONS

LLM-based AI chat models can effectively generate appropriate responses to clinical questions related to bariatric surgery, though the performance of different models can vary greatly. Therefore, caution should be taken when interpreting clinical information provided by LLMs, and clinician oversight is necessary to ensure accuracy. Future investigation is warranted to explore how LLMs might enhance healthcare provision and clinical decision-making in bariatric surgery.

Large Language Model-Based Responses to Patients' In-Basket Messages

Importance

Virtual patient-physician communications have increased since 2020 and negatively impacted primary care physician (PCP) well-being. Generative artificial intelligence (GenAI) drafts of patient messages could potentially reduce health care professional (HCP) workload and improve communication quality, but only if the drafts are considered useful.

Objectives

To assess PCPs' perceptions of GenAI drafts and to examine linguistic characteristics associated with equity and perceived empathy.

Design, Setting, and Participants

This cross-sectional quality improvement study tested the hypothesis that PCPs' ratings of GenAI drafts (created using the electronic health record [EHR] standard prompts) would be equivalent to HCP-generated responses on 3 dimensions. The study was conducted at NYU Langone Health using private patient-HCP communications at 3 internal medicine practices piloting GenAI.

Exposures

Randomly assigned patient messages coupled with either an HCP message or the draft GenAI response.

Main Outcomes and Measures

PCPs rated responses' information content quality (eg, relevance), using a Likert scale, communication quality (eg, verbosity), using a Likert scale, and whether they would use the draft or start anew (usable vs unusable). Branching logic further probed for empathy, personalization, and professionalism of responses. Computational linguistics methods assessed content differences in HCP vs GenAI responses, focusing on equity and empathy.

Results

A total of 16 PCPs (8 [50.0%] female) reviewed 344 messages (175 GenAI drafted; 169 HCP drafted). Both GenAI and HCP responses were rated favorably. GenAI responses were rated higher for communication style than HCP responses (mean [SD], 3.70 [1.15] vs 3.38 [1.20]; P = .01, U = 12 568.5) but were similar to HCPs on information content (mean [SD], 3.53 [1.26] vs 3.41 [1.27]; P = .37; U = 13 981.0) and usable draft proportion (mean [SD], 0.69 [0.48] vs 0.65 [0.47], P = .49, t = -0.6842). Usable GenAI responses were considered more empathetic than usable HCP responses (32 of 86 [37.2%] vs 13 of 79 [16.5%]; difference, 125.5%), possibly attributable to more subjective (mean [SD], 0.54 [0.16] vs 0.31 [0.23]; P < .001; difference, 74.2%) and positive (mean [SD] polarity, 0.21 [0.14] vs 0.13 [0.25]; P = .02; difference, 61.5%) language; they were also numerically longer (mean [SD] word count, 90.5 [32.0] vs 65.4 [62.6]; difference, 38.4%), but the difference was not statistically significant (P = .07) and more linguistically complex (mean [SD] score, 125.2 [47.8] vs 95.4 [58.8]; P = .002; difference, 31.2%).

Conclusions

In this cross-sectional study of PCP perceptions of an EHR-integrated GenAI chatbot, GenAI was found to communicate information better and with more empathy than HCPs, highlighting its potential to enhance patient-HCP communication. However, GenAI drafts were less readable than HCPs', a significant concern for patients with low health or English literacy.

A Nordic Perspective on Patient Online Record Access and the European Health Data Space

The Nordic countries are, together with the United States, forerunners in online record access (ORA), which has now become widespread. The importance of accessible and structured health data has also been highlighted by policy makers internationally. To ensure the full realization of ORA's potential in the short and long term, there is a pressing need to study ORA from a cross-disciplinary, clinical, humanistic, and social sciences perspective that looks beyond strictly technical aspects. In this viewpoint paper, we explore the policy changes in the European Health Data Space (EHDS) proposal to advance ORA across the European Union, informed by our research in a Nordic-led project that carries out the first of its kind, large-scale international investigation of patients' ORA-NORDeHEALTH (Nordic eHealth for Patients: Benchmarking and Developing for the Future). We argue that the EHDS proposal will pave the way for patients to access and control third-party access to their electronic health records. In our analysis of the proposal, we have identified five key principles for ORA: (1) the right to access, (2) proxy access, (3) patient input of their own data, (4) error and omission rectification, and (5) access control. ORA implementation today is fragmented throughout Europe, and the EHDS proposal aims to ensure all European citizens have equal online access to their health data. However, we argue that in order to implement the EHDS, we need more research evidence on the key ORA principles we have identified in our analysis. Results from the NORDeHEALTH project provide some of that evidence, but we have also identified important knowledge gaps that still need further exploration.

Secure Messaging Use Among Patients with Depression: An Analysis Using Real-World Data

Background: Although depression is one of the most common mental health disorders outpacing other diseases and conditions, poor access to care and limited resources leave many untreated. Secure messaging (SM) offers patients an online means to bridge this gap by communicating nonurgent medical questions. We focused on self-care health management behaviors and delved into SM initiation as the initial act of engagement and SM exchanges as continuous engagement patterns. This study examined whether those with depression might be using SM more than those without depression. Methods: Patient portal data were obtained from a large academic medical center's electronic health records spanning 5 years, from January 2018 to December 2022. We organized and analyzed SM initiations and exchanges using the linear mixed-effects modeling technique. Results: Our predictors correlated with SM initiations, accounting for 25.1% of variance explained. In parallel, 24.9% of SM exchanges were attributable to these predictors. Overall, our predictors demonstrate stronger associations with SM exchanges. Discussion: We examined patients with and without depression across 2,629 zip codes over five years. Our findings reveal that the predictors affecting SM initiations and exchanges are multifaceted, with certain predictors enhancing its utilization and others impeding it. Conclusions: SM telehealth service provided support to patients with mental health needs to a greater extent than those without. By increasing access, fostering better communication, and efficiently allocating resources, telehealth services not only encourage patients to begin using SM but also promote sustained interaction through ongoing SM exchanges.

The emerging role of generative artificial intelligence in transplant medicine

Generative artificial intelligence (AI), a subset of machine learning that creates new content based on training data, has witnessed tremendous advances in recent years. Practical applications have been identified in health care in general, and there is significant opportunity in transplant medicine for generative AI to simplify tasks in research, medical education, and clinical practice. In addition, patients stand to benefit from patient education that is more readily provided by generative AI applications. This review aims to catalyze the development and adoption of generative AI in transplantation by introducing basic AI and generative AI concepts to the transplant clinician and summarizing its current and potential applications within the field. We provide an overview of applications to the clinician, researcher, educator, and patient. We also highlight the challenges involved in bringing these applications to the bedside and need for ongoing refinement of generative AI applications to sustainably augment the transplantation field.

Telehealth Trends and Hypertension Management Among Rural and Medicaid Patients After COVID-19

Objective: Examine the associations between rurality and low income with primary care telehealth utilization and hypertension outcomes across multiple years pre- and post-COVID-19 pandemic onset. Methods: We compiled electronic health record data from the mixed rural/urban Dartmouth Health system in New Hampshire, United States, on patients with pre-existing hypertension or diabetes receiving primary care in the period before (January 2018-February 2020) and after the transition period to telehealth during the COVID-19 Pandemic (October 2020-December 2022). Stratifying by rurality and Medicaid enrollment, we examined changes in synchronous (office and telehealth visits, including audio/video use) and asynchronous (patient portal or telephone message) utilization, and control of mean systolic blood pressure (SBP) <140. Results: Analysis included 46,520 patients, of whom 8.2% were Medicaid enrollees, 42.7% urban residents. Telehealth use rates were 12% for rural versus 6.4% for urban, and 15% for Medicaid versus 8.4% non-Medicaid. The overall postpandemic telehealth visit rate was 0.29 per patient per year. Rural patients had a larger increase in telehealth use (additional 0.21 per year, 95% CI, 0.19-0.23) compared with urban, as did Medicaid (0.32, 95% CI 0.29-0.36) compared with non-Medicaid. Among the 38,437 patients with hypertension, SBP control worsened from 83% to 79% of patients across periods. In multivariable analysis, rurality corresponded to worsened control rates compared with urban (additional 2.4% decrease, 95% CI 2.1-2.8%); Medicaid and telehealth use were not associated with worsened control. Conclusions: Telehealth expansion enabled a higher shift to telehealth for rural and low-income patients without impairing hypertension management.

Leveraging large language models for generating responses to patient messages-a subjective analysis

OBJECTIVE

This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal.

MATERIALS AND METHODS

Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness.

RESULTS

The dataset consisted of 499 794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider's responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT's responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness.

CONCLUSION

This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.

Association of Patient Portal Messaging with Survival Among Radiation Oncology Patients

PURPOSE

The shift to electronic health records has led to both patient portal messaging and large amounts of digital, real-world data for research. The objective of this study was to examine the association between portal messaging and survival among radiation oncology patients, using real-world data.

METHODS AND MATERIALS

This retrospective cohort study included patients at least 21 years old and seen by radiation oncology providers between January 14, 2014, and April 23, 2023, at the University of California, San Francisco. We developed Cox proportional hazards models for the outcome of death and examined factors associated with portal messaging using logistic regression models.

RESULTS

Among 25,367 patients, the median age was 64 (interquartile range [IR], 54-72), 13,175 (52%) were White, and 14,389 (57%) were male. Overall, as the first message in a thread, 8986 (35%) patients sent messages to radiation oncology providers, and 4218 (17%) patients were sent messages from radiation oncology providers. Patients with head and neck or genitourinary malignancies were more likely than those with other diagnoses to send portal messages to and be sent portal messages from radiation oncology providers. Both sending portal messages to radiation oncology providers (hazard ratio [HR], 0.90; 95% confidence interval [CI], 0.84-0.96; P = .001) and being sent messages from radiation oncology providers (HR, 0.77; CI, 0.70-0.84; P < .001) as the first message in a thread were associated with patient survival after adjusting for socioeconomic, disease, and treatment characteristics. There were disparities among patients sending portal messages to radiation oncology providers, including for Black versus White patients (odds ratio [OR], 0.60; CI, 0.51-0.69; P < .001) and for Medicaid versus Medicare patients (OR, 0.70; CI, 0.62-0.79; P < .001). There were also disparities among patients being sent portal messages by radiation oncology providers, including for Black versus White patients (OR, 0.77; CI, 0.64-0.91; P = .003), for Medicaid versus Medicare patients (OR, 0.76; CI, 0.65-0.89; P < .001), and for patients with female versus male providers (OR, 1.47; CI 1.34-1.62; P < .001).

CONCLUSIONS

Sending portal messages to and being sent portal messages from radiation oncology providers were associated with better survival. Future studies should elucidate how best to support patient and provider engagement.

Virtual Scribes and Physician Time Spent on Electronic Health Records

Importance

Time on the electronic health record (EHR) is associated with burnout among physicians. Newer virtual scribe models, which enable support from either a real-time or asynchronous scribe, have the potential to reduce the burden of the EHR and EHR-related documentation.

Objective

To characterize the association of use of virtual scribes with changes in physicians' EHR time and note and order composition and to identify the physician, scribe, and scribe response factors associated with changes in EHR time upon virtual scribe use.

Design, Setting, and Participants

Retrospective, pre-post quality improvement study of 144 physicians across specialties who had used a scribe for at least 3 months from January 2020 to September 2022, were affiliated with Brigham and Women's Hospital and Massachusetts General Hospital, and cared for patients in the outpatient setting. Data were analyzed from November 2022 to January 2024.

Exposure

Use of either a real-time or asynchronous virtual scribe.

Main Outcomes

Total EHR time, time on notes, and pajama time (5:30 pm to 7:00 am on weekdays and nonscheduled weekends and holidays), all per appointment; proportion of the note written by the physician and team contribution to orders.

Results

The main study sample included 144 unique physicians who had used a virtual scribe for at least 3 months in 152 unique scribe participation episodes (134 [88.2%] had used an asynchronous scribe service). Nearly two-thirds of the physicians (91 physicians [63.2%]) were female and more than half (86 physicians [59.7%]) were in primary care specialties. Use of a virtual scribe was associated with significant decreases in total EHR time per appointment (mean [SD] of 5.6 [16.4] minutes; P < .001) in the 3 months after vs the 3 months prior to scribe use. Scribe use was also associated with significant decreases in note time per appointment and pajama time per appointment (mean [SD] of 1.3 [3.3] minutes; P < .001 and 1.1 [4.0] minutes; P = .004). In a multivariable linear regression model, the following factors were associated with significant decreases in total EHR time per appointment with a scribe use at 3 months: practicing in a medical specialty (-7.8; 95% CI, -13.4 to -2.2 minutes), greater baseline EHR time per appointment (-0.3; 95% CI, -0.4 to -0.2 minutes per additional minute of baseline EHR time), and decrease in the percentage of the note contributed by the physician (-9.1; 95% CI, -17.3 to -0.8 minutes for every percentage point decrease).

Conclusions and Relevance

In 2 academic medical centers, use of virtual scribes was associated with significant decreases in total EHR time, time spent on notes, and pajama time, all per appointment. Virtual scribes may be particularly effective among medical specialists and those physicians with greater baseline EHR time.

Billing for Electronic Patient-Physician Communications: An Ethical Analysis

This review paper analyzes the ethical implications of billing patients for electronic communication with physicians through electronic health records, a practice already adopted by medical institutions such as the Cleveland Clinic. The analysis assesses how billing aligns with pillars of medical ethics which include beneficence, respect for persons, and justice. Although billing may enhance communication, improve patient care, and alleviate physician burnout, concerns arise over potential consequences on patient autonomy, trust, and health care disparities. The review delves into the intricate balance of these ethical principles by first considering the potential benefits of incentivizing concise questions and improving physician workload management through billing. By reducing messages, this approach can potentially mitigate burnout and enhance care. It also acknowledges potential drawbacks such as deterring patients because of financial constraints and eroding trust in physicians and the medical team. It emphasizes the necessity of thoroughly examining all aspects of this intricate ethical dilemma to formulate a nuanced solution that protects patient well-being while respecting physicians. We propose a middle-ground approach involving nominal and transparent billing on the basis of the question's complexity, urgency, and level of expertise required in the response. Transparent billing policies, up-front communication of costs, and potential fee waivers on the basis of socioeconomic status can address equity concerns and maintain patient trust. Striking a balance between the potential benefits and drawbacks of billing for patient questions is crucial in maintaining ethical patient-physician interactions and equitable health care provision. The analysis underscores the importance of aligning online patient-physician communication with ethical principles within the evolving digital health care landscape.

Predicting Primary Care Physician Burnout From Electronic Health Record Use Measures

OBJECTIVE

To evaluate the ability of routinely collected electronic health record (EHR) use measures to predict clinical work units at increased risk of burnout and potentially most in need of targeted interventions.

METHODS

In this observational study of primary care physicians, we compiled clinical workload and EHR efficiency measures, then linked these measures to 2 years of well-being surveys (using the Stanford Professional Fulfillment Index) conducted from April 1, 2019, through October 16, 2020. Physicians were grouped into training and confirmation data sets to develop predictive models for burnout. We used gradient boosting classifier and other prediction modeling algorithms to quantify the predictive performance by the area under the receiver operating characteristics curve (AUC).

RESULTS

Of 278 invited physicians from across 60 clinics, 233 (84%) completed 396 surveys. Physicians were 67% women with a median age category of 45 to 49 years. Aggregate burnout score was in the high range (≥3.325/10) on 111 of 396 (28%) surveys. Gradient boosting classifier of EHR use measures to predict burnout achieved an AUC of 0.59 (95% CI, 0.48 to 0.77) and an area under the precision-recall curve of 0.29 (95% CI, 0.20 to 0.66). Other models' confirmation set AUCs ranged from 0.56 (random forest) to 0.66 (penalized linear regression followed by dichotomization). Among the most predictive features were physician age, team member contributions to notes, and orders placed with user-defined preferences. Clinic-level aggregate measures identified the top quartile of clinics with 56% sensitivity and 85% specificity.

CONCLUSION

In a sample of primary care physicians, routinely collected EHR use measures demonstrated limited ability to predict individual burnout and moderate ability to identify high-risk clinics.

Content of Patient Electronic Messages to Physicians in a Large Integrated System

This quality improvement study describes the content of electronic health record messages from patients to physicians in a large integrated health care system using natural language processing algorithms.

Quality and Dependability of ChatGPT and DingXiangYuan Forums for Remote Orthopedic Consultations: Comparative Analysis

BACKGROUND

The widespread use of artificial intelligence, such as ChatGPT (OpenAI), is transforming sectors, including health care, while separate advancements of the internet have enabled platforms such as China's DingXiangYuan to offer remote medical services.

OBJECTIVE

This study evaluates ChatGPT-4's responses against those of professional health care providers in telemedicine, assessing artificial intelligence's capability to support the surge in remote medical consultations and its impact on health care delivery.

METHODS

We sourced remote orthopedic consultations from "Doctor DingXiang," with responses from its certified physicians as the control and ChatGPT's responses as the experimental group. In all, 3 blindfolded, experienced orthopedic surgeons assessed responses against 7 criteria: "logical reasoning," "internal information," "external information," "guiding function," "therapeutic effect," "medical knowledge popularization education," and "overall satisfaction." We used Fleiss κ to measure agreement among multiple raters.

RESULTS

Initially, consultation records for a cumulative count of 8 maladies (equivalent to 800 cases) were gathered. We ultimately included 73 consultation records by May 2023, following primary and rescreening, in which no communication records containing private information, images, or voice messages were transmitted. After statistical scoring, we discovered that ChatGPT's "internal information" score (mean 4.61, SD 0.52 points vs mean 4.66, SD 0.49 points; P=.43) and "therapeutic effect" score (mean 4.43, SD 0.75 points vs mean 4.55, SD 0.62 points; P=.32) were lower than those of the control group, but the differences were not statistically significant. ChatGPT showed better performance with a higher "logical reasoning" score (mean 4.81, SD 0.36 points vs mean 4.75, SD 0.39 points; P=.38), "external information" score (mean 4.06, SD 0.72 points vs mean 3.92, SD 0.77 points; P=.25), and "guiding function" score (mean 4.73, SD 0.51 points vs mean 4.72, SD 0.54 points; P=.96), although the differences were not statistically significant. Meanwhile, the "medical knowledge popularization education" score of ChatGPT was better than that of the control group (mean 4.49, SD 0.67 points vs mean 3.87, SD 1.01 points; P<.001), and the difference was statistically significant. In terms of "overall satisfaction," the difference was not statistically significant between the groups (mean 8.35, SD 1.38 points vs mean 8.37, SD 1.24 points; P=.92). According to how Fleiss κ values were interpreted, 6 of the control group's score points were classified as displaying "fair agreement" (P<.001), and 1 was classified as showing "substantial agreement" (P<.001). In the experimental group, 3 points were classified as indicating "fair agreement," while 4 suggested "moderate agreement" (P<.001).

CONCLUSIONS

ChatGPT-4 matches the expertise found in DingXiangYuan forums' paid consultations, excelling particularly in scientific education. It presents a promising alternative for remote health advice. For health care professionals, it could act as an aid in patient education, while patients may use it as a convenient tool for health inquiries.

Differences in Care Team Response to Patient Portal Messages by Patient Race and Ethnicity

Importance

The COVID-19 pandemic was associated with substantial growth in patient portal messaging. Higher message volumes have largely persisted, reflecting a new normal. Prior work has documented lower message use by patients who belong to minoritized racial and ethnic groups, but research has not examined differences in care team response to messages. Both have substantial ramifications on resource allocation and care access under a new care paradigm with portal messaging as a central channel for patient-care team communication.

Objective

To examine differences in how care teams respond to patient portal messages sent by patients from different racial and ethnic groups.

Design, Setting, and Participants

In a cross-sectional design in a large safety-net health system, response outcomes from medical advice message threads sent from January 1, 2021, through November 24, 2021, from Asian, Black, Hispanic, and White patients were compared, controlling for patient and message thread characteristics. Asian, Black, Hispanic, and White patients with 1 or more adult primary care visits at Boston Medical Center in calendar year 2020 were included. Data analysis was conducted from June 23, 2022, through December 21, 2023.

Exposure

Patient race and ethnicity.

Main Outcomes and Measures

Rates at which medical advice request messages were responded to by care teams and the types of health care professionals that responded.

Results

A total of 39 043 patients were included in the sample: 2006 were Asian, 21 600 were Black, 7185 were Hispanic, and 8252 were White. A total of 22 744 (58.3%) patients were women and mean (SD) age was 50.4 (16.7) years. In 2021, these patients initiated 57 704 medical advice request message threads. When patients who belong to minoritized racial and ethnic groups sent these messages, the likelihood of receiving any care team response was similar, but the types of health care professionals that responded differed. Black patients were 3.95 percentage points (pp) less likely (95% CI, -5.34 to -2.57 pp; P < .001) to receive a response from an attending physician, and 3.01 pp more likely (95% CI, 1.76-4.27 pp; P < .001) to receive a response from a registered nurse, corresponding to a 17.4% lower attending response rate. Similar, but smaller, differences were observed for Asian and Hispanic patients.

Conclusions and Relevance

The findings of this study suggest lower prioritization of patients who belong to minoritized racial and ethnic groups during triaging. Understanding and addressing these disparities will be important for improving care equity and informing health care delivery support algorithms.

Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages

Importance

The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making.

Objective

To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox.

Design, Setting, and Participants

A 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot.

Intervention

Draft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act-compliant electronic health record-integrated large language model.

Main Outcomes and Measures

The primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey.

Results

A total of 197 clinicians were enrolled in the pilot; 35 clinicians who were prepilot beta users, out of office, or not tied to a specific ambulatory clinic were excluded, leaving 162 clinicians included in the analysis. The survey analysis cohort consisted of 73 participants (45.1%) who completed both the presurvey and postsurvey. In gastroenterology and hepatology, there were 58 physicians and APPs and 10 nurses. In primary care, there were 83 physicians and APPs, 4 nurses, and 8 clinical pharmacists. The mean AI-generated draft response utilization rate across clinicians was 20%. There was no change in reply action time, write time, or read time between the prepilot and pilot periods. There were statistically significant reductions in the 4-item physician task load score derivative (mean [SD], 61.31 [17.23] presurvey vs 47.26 [17.11] postsurvey; paired difference, -13.87; 95% CI, -17.38 to -9.50; P < .001) and work exhaustion scores (mean [SD], 1.95 [0.79] presurvey vs 1.62 [0.68] postsurvey; paired difference, -0.33; 95% CI, -0.50 to -0.17; P < .001).

Conclusions and Relevance

In this quality improvement study of an early implementation of generative AI, there was notable adoption, usability, and improvement in assessments of burden and burnout. There was no improvement in time. Further code-to-bedside testing is needed to guide future development and organizational strategy.

Patient Portal Message Volume and Time Spent on the EHR: an Observational Study of Primary Care Clinicians

BACKGROUND

As patient-initiated messaging rises, identifying variation in message volume and its relationship to clinician workload is essential.

OBJECTIVE

To describe the association between variation in message volume over time and time spent on the electronic health record (EHR) outside of scheduled hours.

DESIGN

Retrospective cohort study.

PARTICIPANTS

Primary care clinicians at Cleveland Clinic Health System.

MAIN MEASURES

We categorized clinicians according to their number of quarterly incoming medical advice messages (i.e., message volume) between January 2019 and December 2021 using group-based trajectory modeling. We assessed change in quarterly messages and outpatient visits between October-December 2019 (Q4) and October-December 2021 (Q12). The primary outcome was time outside of scheduled hours spent on the EHR. We used mixed effects logistic regression to describe the association between incoming portal messages and time spent on the EHR by clinician messaging group and at the clinician level.

KEY RESULTS

Among the 150 clinicians, 31% were in the low-volume group (206 messages per quarter per clinician), 47% were in the moderate-volume group (505 messages), and 22% were in the high-volume group (840 messages). Mean quarterly messages increased from 340 to 695 (p < 0.001) between Q4 and Q12; mean quarterly outpatient visits fell from 711 to 575 (p = 0.005). While time spent on the EHR outside of scheduled hours increased modestly for all clinicians, this did not significantly differ by message group. Across all clinicians, each additional 10 messages was associated with an average of 12 min per quarter of additional time spent on the EHR (p < 0.001).

CONCLUSIONS

Message volume increased substantially over the study period and varied by group. While messages were associated with additional time spent on the EHR outside of scheduled hours, there was no significant difference in time spent on the EHR between the high and low message volume groups.

Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross-Sectional Study

OBJECTIVE

The objective of the current study was to assess the quality of large language model (LLM) chatbot versus physician-generated responses to patient-generated rheumatology questions.

METHODS

We conducted a single-center cross-sectional survey of rheumatology patients (n = 17) in Edmonton, Alberta, Canada. Patients evaluated LLM chatbot versus physician-generated responses for comprehensiveness and readability, with four rheumatologists also evaluating accuracy by using a Likert scale from 1 to 10 (1 being poor, 10 being excellent).

RESULTS

Patients rated no significant difference between artificial intelligence (AI) and physician-generated responses in comprehensiveness (mean 7.12 ± SD 0.99 vs 7.52 ± 1.16; P = 0.1962) or readability (7.90 ± 0.90 vs 7.80 ± 0.75; P = 0.5905). Rheumatologists rated AI responses significantly poorer than physician responses on comprehensiveness (AI 5.52 ± 2.13 vs physician 8.76 ± 1.07; P < 0.0001), readability (AI 7.85 ± 0.92 vs physician 8.75 ± 0.57; P = 0.0003), and accuracy (AI 6.48 ± 2.07 vs physician 9.08 ± 0.64; P < 0.0001). The proportion of preference to AI- versus physician-generated responses by patients and physicians was 0.45 ± 0.18 and 0.15 ± 0.08, respectively (P = 0.0106). After learning that one answer for each question was AI generated, patients were able to correctly identify AI-generated answers at a lower proportion compared to physicians (0.49 ± 0.26 vs 0.97 ± 0.04; P = 0.0183). The average word count of AI answers was 69.10 ± 25.35 words, as compared to 98.83 ± 34.58 words for physician-generated responses (P = 0.0008).

CONCLUSION

Rheumatology patients rated AI-generated responses to patient questions similarly to physician-generated responses in terms of comprehensiveness, readability, and overall preference. However, rheumatologists rated AI responses significantly poorer than physician-generated responses, suggesting that LLM chatbot responses are inferior to physician responses, a difference that patients may not be aware of.

Guidance for reporting analyses of metadata on electronic health record use

INTRODUCTION

Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies.

RECOMMENDATIONS

In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

Quantifying the impact of telemedicine and patient medical advice request messages on physicians' work-outside-work

The COVID-19 pandemic has boosted digital health utilization, raising concerns about increased physicians' after-hours clinical work ("work-outside-work"). The surge in patients' digital messages and additional time spent on work-outside-work by telemedicine providers underscores the need to evaluate the connection between digital health utilization and physicians' after-hours commitments. We examined the impact on physicians' workload from two types of digital demands - patients' messages requesting medical advice (PMARs) sent to physicians' inbox (inbasket), and telemedicine. Our study included 1716 ambulatory-care physicians in New York City regularly practicing between November 2022 and March 2023. Regression analyses assessed primary and interaction effects of (PMARs) and telemedicine on work-outside-work. The study revealed a significant effect of PMARs on physicians' work-outside-work and that this relationship is moderated by physicians' specialties. Non-primary care physicians or specialists experienced a more pronounced effect than their primary care peers. Analysis of their telemedicine load revealed that primary care physicians received fewer PMARs and spent less time in work-outside-work with more telemedicine. Specialists faced increased PMARs and did more work-outside-work as telemedicine visits increased which could be due to the difference in patient panels. Reducing PMAR volumes and efficient inbasket management strategies needed to reduce physicians' work-outside-work. Policymakers need to be cognizant of potential disruptions in physicians carefully balanced workload caused by the digital health services.

National Trends in Billing Secure Messages as E-Visits

This study assesses US trends in e-visit billing using national all-payer claims.

Navigating the doctor-patient-AI relationship - a mixed-methods study of physician attitudes toward artificial intelligence in primary care

BACKGROUND

Artificial intelligence (AI) is a rapidly advancing field that is beginning to enter the practice of medicine. Primary care is a cornerstone of medicine and deals with challenges such as physician shortage and burnout which impact patient care. AI and its application via digital health is increasingly presented as a possible solution. However, there is a scarcity of research focusing on primary care physician (PCP) attitudes toward AI. This study examines PCP views on AI in primary care. We explore its potential impact on topics pertinent to primary care such as the doctor-patient relationship and clinical workflow. By doing so, we aim to inform primary care stakeholders to encourage successful, equitable uptake of future AI tools. Our study is the first to our knowledge to explore PCP attitudes using specific primary care AI use cases rather than discussing AI in medicine in general terms.

METHODS

From June to August 2023, we conducted a survey among 47 primary care physicians affiliated with a large academic health system in Southern California. The survey quantified attitudes toward AI in general as well as concerning two specific AI use cases. Additionally, we conducted interviews with 15 survey respondents.

RESULTS

Our findings suggest that PCPs have largely positive views of AI. However, attitudes often hinged on the context of adoption. While some concerns reported by PCPs regarding AI in primary care focused on technology (accuracy, safety, bias), many focused on people-and-process factors (workflow, equity, reimbursement, doctor-patient relationship).

CONCLUSION

Our study offers nuanced insights into PCP attitudes towards AI in primary care and highlights the need for primary care stakeholder alignment on key issues raised by PCPs. AI initiatives that fail to address both the technological and people-and-process concerns raised by PCPs may struggle to make an impact.

More Tethered to the EHR: EHR Workload Trends Among Academic Primary Care Physicians, 2019-2023

PURPOSE

The purpose of this study is to evaluate recent trends in primary care physician (PCP) electronic health record (EHR) workload.

METHODS

This longitudinal study observed the EHR use of 141 academic PCPs over 4 years (May 2019 to March 2023). Ambulatory full-time equivalency (aFTE), visit volume, and panel size were evaluated. Electronic health record time and inbox message volume were measured per 8 hours of scheduled clinic appointments.

RESULTS

From the pre-COVID-19 pandemic year (May 2019 to February 2020) to the most recent study year (April 2022 to March 2023), the average time PCPs spent in the EHR per 8 hours of scheduled clinic appointments increased (+28.4 minutes, 7.8%), as did time in orders (+23.1 minutes, 58.9%), inbox (+14.0 minutes, 24.4%), chart review (+7.2 minutes, 13.0%), notes (+2.9 minutes, 2.3%), outside scheduled hours on days with scheduled appointments (+6.4 minutes, 8.2%), and on unscheduled days (+13.6 minutes, 19.9%). Primary care physicians received more patient medical advice requests (+5.4 messages, 55.5%) and prescription messages (+2.3, 19.5%) per 8 hours of scheduled clinic appointments, but fewer patient calls (-2.8, -10.5%) and results messages (-0.3, -2.7%). While total time in the EHR continued to increase in the final study year (+7.7 minutes, 2.0%), inbox time decreased slightly from the year prior (-2.2 minutes, -3.0%). Primary care physicians' average aFTE decreased 5.2% from 0.66 to 0.63 over 4 years.

CONCLUSIONS

Primary care physicians' time in the EHR continues to grow. While PCPs' inbox time may be stabilizing, it is still substantially higher than pre-pandemic levels. It is imperative health systems develop strategies to change the EHR workload trajectory to minimize PCPs' occupational stress and mitigate unnecessary reductions in effective physician workforce resulting from the increased EHR burden.

Online artificial intelligence platforms and their applicability to gastrointestinal surgical operations

BACKGROUND

The internet is a common source of health information for patients. Interactive online artificial intelligence (AI) may be a more reliable source of health-related information than traditional search engines. This study aimed to assess the quality and perceived utility of chat-based AI responses related to 3 common gastrointestinal (GI) surgical procedures.

METHODS

A survey of 24 questions covering general perioperative information on cholecystectomy, pancreaticoduodenectomy (PD), and colectomy was created. Each question was posed to Chat Generative Pre-trained Transformer (ChatGPT) in June 2023, and the generated responses were recorded. The quality and perceived utility of responses were independently and subjectively graded by expert respondents specific to each surgical field. Grades were classified as "poor," "fair," "good," "very good," or "excellent."

RESULTS

Among the 45 respondents (general surgeon [n = 13], surgical oncologist [n = 18], colorectal surgeon [n = 13], and transplant surgeon [n = 1]), most practiced at an academic facility (95.6%). Respondents had been in practice for a mean of 12.3 years (general surgeon, 14.5 ± 7.2; surgical oncologist, 12.1 ± 8.2; colorectal surgeon, 10.2 ± 8.0) and performed a mean 53 index operations annually (cholecystectomy, 47 ± 28; PD, 28 ± 27; colectomy, 81 ± 44). Overall, the most commonly assigned quality grade was "fair" or "good" for most responses (n = 622/1080, 57.6%). Most of the 1080 total utility grades were "fair" (n = 279, 25.8%) or "good" (n = 344, 31.9%), whereas only 129 utility grades (11.9%) were "poor." Of note, ChatGPT responses related to cholecystectomy (45.3% ["very good"/"excellent"] vs 18.1% ["poor"/"fair"]) were deemed to be better quality than AI responses about PD (18.9% ["very good"/"excellent"] vs 46.9% ["poor"/"fair"]) or colectomy (31.4% ["very good"/"excellent"] vs 38.3% ["poor"/"fair"]). Overall, only 20.0% of the experts deemed ChatGPT to be an accurate source of information, whereas 15.6% of the experts found it unreliable. Moreover, 1 in 3 surgeons deemed ChatGPT responses as not likely to reduce patient-physician correspondence (31.1%) or not comparable to in-person surgeon responses (35.6%).

CONCLUSIONS

Although a potential resource for patient education, ChatGPT responses to common GI perioperative questions were deemed to be of only modest quality and utility to patients. In addition, the relative quality of AI responses varied markedly on the basis of procedure type.

Use of Electronic Medical Records (EMR) in Gerontology: Benefits, Considerations and a Promising Future

Electronic medical records (EMRs) have many benefits in clinical research in gerontology, enabling data analysis, development of prognostic tools and disease risk prediction. EMRs also offer a range of advantages in clinical practice, such as comprehensive medical records, streamlined communication with healthcare providers, remote data access, and rapid retrieval of test results, ultimately leading to increased efficiency, enhanced patient safety, and improved quality of care in gerontology, which includes benefits like reduced medication use and better patient history taking and physical examination assessments. The use of artificial intelligence (AI) and machine learning (ML) approaches on EMRs can further improve disease diagnosis, symptom classification, and support clinical decision-making. However, there are also challenges related to data quality, data entry errors, as well as the ethics and safety of using AI in healthcare. This article discusses the future of EMRs in gerontology and the application of AI and ML in clinical research. Ethical and legal issues surrounding data sharing and the need for healthcare professionals to critically evaluate and integrate these technologies are also emphasized. The article concludes by discussing the challenges related to the use of EMRs in research as well as in their primary intended use, the daily clinical practice.

Changes in Physician Electronic Health Record Use With the Expansion of Telemedicine

Importance

Understanding the drivers of electronic health record (EHR) burden, including EHR time and patient messaging, may directly inform strategies to address physician burnout. Given the COVID-19-induced expansion of telemedicine-now used for a substantial proportion of ambulatory encounters-its association with EHR burden should be evaluated.

Objective

To measure the association of the telemedicine expansion with time spent working in the EHR and with patient messaging among ambulatory physicians before and after the onset of the COVID-19 pandemic.

Design, Setting, and Participants

This longitudinal cohort study analyzed weekly EHR metadata of ambulatory physicians at UCSF Health, a large academic medical center. The same EHR measures were compared for 1 year before the COVID-19 pandemic (August 2018-September 2019) with the same period 1 year after its onset (August 2020-September 2021). Multivariable regression models evaluating the association between level of telemedicine use and EHR use were then assessed after the onset of the pandemic. The sample included all physician-weeks with at least 1 scheduled half-day clinic in the 11 largest ambulatory specialties at UCSF Health. Data analyses were performed from March 1, 2022, through July 1, 2023.

Exposures

Physicians' weekly modality mix of either entirely face-to-face visits, mixed modalities, or entirely telemedicine.

Main Outcomes and Measures

The EHR time during and outside of patient scheduled hours (PSHs), time spent documenting (normalized per 8 PSHs), and electronic messages sent to and received from patients.

Results

The study sample included 1052 physicians (437 [41.5%] men and 615 [58.5%] women) during 115 weeks, which provided 35 697 physician-week observations. Comparing the period before to the period after pandemic onset showed that physician time spent working in the EHR during PSHs increased from 4.53 to 5.46 hours per 8 PSH (difference, 0.93; 95% CI, 0.87-0.98; P < 0.001); outside of PSHs, increased from 4.29 to 5.34 hours (difference, 1.04; 95% CI, 0.95-1.14; P < 0.001); and time documenting during and outside of PSHs increased from 6.35 to 8.18 hours (difference, 1.83; 95% CI, 1.72-1.94; P < 0.001). Mean weekly messages received from patients increased from 16.76 to 30.33, and messages sent to patients increased from 13.82 to 29.83. In multivariable models, weeks with a mix of face-to-face and telemedicine (β, 0.43; 95% CI, 0.31-0.55; P < .001) visits or entirely telemedicine (β, 0.91; 95% CI, 0.74-1.09; P < .001) had more EHR time during PSHs than all face-to-face weeks, with similar results for EHR time outside of PSHs. There was no association between telemedicine use and messages received from patients, whereas mixed modalities (β, -0.90; 95% CI, -1.73 to -0.08; P = .03) and all telemedicine (β, -4.06; 95% CI, -5.19 to -2.93; P < .001) were associated with fewer messages sent to patients compared with entirely face-to-face weeks.

Conclusions and Relevance

The findings of this longitudinal cohort study suggest that telemedicine is associated with greater physician time spent working in the EHR, both during and outside of scheduled hours, mostly documenting visits and not messaging patients. Health systems may need to adjust productivity expectations for physicians and develop strategies to address EHR documentation burden for physicians.

Assessing the utility of ChatGPT as an artificial intelligence-based large language model for information to answer questions on myopia

PURPOSE

ChatGPT is an artificial intelligence language model, which uses natural language processing to simulate human conversation. It has seen a wide range of applications including healthcare education, research and clinical practice. This study evaluated the accuracy of ChatGPT in providing accurate and quality information to answer questions on myopia.

METHODS

A series of 11 questions (nine categories of general summary, cause, symptom, onset, prevention, complication, natural history, treatment and prognosis) were generated for this cross-sectional study. Each question was entered five times into fresh ChatGPT sessions (free from influence of prior questions). The responses were evaluated by a five-member team of optometry teaching and research staff. The evaluators individually rated the accuracy and quality of responses on a Likert scale, where a higher score indicated greater quality of information (1: very poor; 2: poor; 3: acceptable; 4: good; 5: very good). Median scores for each question were estimated and compared between evaluators. Agreement between the five evaluators and the reliability statistics of the questions were estimated.

RESULTS

Of the 11 questions on myopia, ChatGPT provided good quality information (median scores: 4.0) for 10 questions and acceptable responses (median scores: 3.0) for one question. Out of 275 responses in total, 66 (24%) were rated very good, 134 (49%) were rated good, whereas 60 (22%) were rated acceptable, 10 (3.6%) were rated poor and 5 (1.8%) were rated very poor. Cronbach's α of 0.807 indicated good level of agreement between test items. Evaluators' ratings demonstrated 'slight agreement' (Fleiss's κ, 0.005) with a significant difference in scoring among the evaluators (Kruskal-Wallis test, p < 0.001).

CONCLUSION

Overall, ChatGPT generated good quality information to answer questions on myopia. Although ChatGPT shows great potential in rapidly providing information on myopia, the presence of inaccurate responses demonstrates that further evaluation and awareness concerning its limitations are crucial to avoid potential misinterpretation.

Ways to Improve Workflow and Morale in an Ophthalmology Clinic: Survey Advice from Clinic Staff

Objective

We aim to improve job workflow and satisfaction amongst clinic staff at an academic ophthalmology department.

Methods

We analyzed survey data given over a 2-week period in July 2021. The participants were support staff (N = 18) from an academic ophthalmology department. Paper surveys were distributed to participants and returned anonymously for analysis. Results: The survey contained 9 Likert-style categorical questions, 2 of which were free response options. A total of 22 participants attempted the survey, 18 of these (82%) were complete and included in analysis. About half of the staff were satisfied with the current workflow 10/18 (56%). Staff who were clinical care coordinators had the lowest average satisfaction (2/5 on a 5-point scale) and the nursing team had the highest average (4.75/5). The most common staff suggestion for improving workflow efficiency was to train residents on forwarding and answering messages more effectively.

Conclusion

This survey suggests that assigning patient message processing to the nursing staff can improve job satisfaction and workflow. Staff told us that the most exciting part of the job was appreciation from coworkers 9/30 (30%) and from physicians 8/30 (27%). The findings provide advice to physicians for optimizing communication, and staff experience, within their own ophthalmology clinics.

Association of physician burnout with perceived EHR work stress and potentially actionable factors

OBJECTIVE

Physicians of all specialties experienced unprecedented stressors during the COVID-19 pandemic, exacerbating preexisting burnout. We examine burnout's association with perceived and actionable electronic health record (EHR) workload factors and personal, professional, and organizational characteristics with the goal of identifying levers that can be targeted to address burnout.

MATERIALS AND METHODS

Survey of physicians of all specialties in an academic health center, using a standard measure of burnout, self-reported EHR work stress, and EHR-based work assessed by the number of messages regarding prescription reauthorization and use of a staff pool to triage messages. Descriptive and multivariable regression analyses examined the relationship among burnout, perceived EHR work stress, and actionable EHR work factors.

RESULTS

Of 1038 eligible physicians, 627 responded (60% response rate), 49.8% reported burnout symptoms. Logistic regression analysis suggests that higher odds of burnout are associated with physicians feeling higher level of EHR stress (odds ratio [OR], 1.15; 95% confidence interval [CI], 1.07-1.25), having more prescription reauthorization messages (OR, 1.23; 95% CI, 1.04-1.47), not feeling valued (OR, 3.38; 95% CI, 1.69-7.22) or aligned in values with clinic leaders (OR, 2.81; 95% CI, 1.87-4.27), in medical practice for ≤15 years (OR, 2.57; 95% CI, 1.63-4.12), and sleeping for <6 h/night (OR, 1.73; 95% CI, 1.12-2.67).

DISCUSSION

Perceived EHR stress and prescription reauthorization messages are significantly associated with burnout, as are non-EHR factors such as not feeling valued or aligned in values with clinic leaders. Younger physicians need more support.

CONCLUSION

A multipronged approach targeting actionable levers and supporting young physicians is needed to implement sustainable improvements in physician well-being.

COVID exacerbated the gender disparity in physician electronic health record inbox burden

The COVID-19 pandemic was associated with significant changes to the delivery of ambulatory care, including a dramatic increase in patient messages to physicians. While asynchronous messaging is a valuable communication modality for patients, a greater volume of patient messages is associated with burnout and decreased well-being for physicians. Given that women physicians experienced greater electronic health record (EHR) burden and received more patient messages pre-pandemic, there is concern that COVID may have exacerbated this disparity. Using EHR audit log data of ambulatory physicians at an academic medical center, we used a difference-in-differences framework to evaluate the impact of the pandemic on patient message volume and compare differences between men and women physicians. We found patient message volume increased post-COVID for all physicians, and women physicians saw an additional increase compared to men. Our results contribute to the growing evidence of different communication expectations for women physicians that contribute to the gender disparity in EHR burden.

Visit and Between-Visit Interaction Frequency Before and After COVID-19 Telehealth Implementation

Importance

Telehealth implementation associated with the COVID-19 public health emergency (PHE) affected patient-clinical team interactions in numerous ways. Yet, studies have narrowly examined billed patient-clinician visits rather than including visits with other team members (eg, pharmacists) or between-visit interactions.

Objective

To evaluate rates of change over time in visits (in-person, telehealth) and between-visit interactions (telephone calls, patient portal messages) overall and by key patient characteristics.

Design, Setting, and Participants

This retrospective cohort study included adults with diabetes receiving primary care at urban academic (University of California San Francisco [UCSF]) and safety-net (San Francisco Health Network [SFHN]) health care systems. Encounters from April 2019 to March 2021 were analyzed.

Exposure

Telehealth implementation over 3 periods: pre-PHE (April 2019 to March 2020), strict shelter-in-place (April to June 2020), and hybrid-PHE (July 2020 to March 2021).

Main Outcomes and Measures

The main outcomes were rates of change in monthly mean number of total encounters, visits with any health care team member, visits with billing clinicians, and between-visit interactions. Key patient-level characteristics were age, race and ethnicity, language, and neighborhood socioeconomic status (nSES).

Results

Of 15 148 patients (4976 UCSF; 8975 SFHN) included, 2464 (16%) were 75 years or older, 7734 (51%) were female patients, 9823 (65%) self-identified as racially or ethnically minoritized, 6223 (41%) had a non-English language preference, and 4618 (31%) lived in the lowest nSES quintile. After accounting for changes to care delivery through an interrupted time-series analysis, total encounters increased in the hybrid-PHE period (UCSF: 2.3% per patient/mo; 95% CI, 1.6%-2.9% per patient/mo; SFHN: 1.8% per patient/mo, 95% CI, 1.3%-2.2% per patient/mo), associated primarily with growth in between-visit interactions (UCSF: 3.1% per patient/mo, 95% CI, 2.3%-3.8% per patient/mo; SFHN: 2.9% per patient/mo, 95% CI, 2.3%-3.4% per patient/mo). In contrast, rates of visits were stable during the hybrid-PHE period. Although there were fewer differences in visit use by key patient-level characteristics during the hybrid-PHE period, pre-PHE differences in between-visit interactions persisted during the hybrid-PHE period at SFHN. Asian and Chinese-speaking patients at SFHN had fewer monthly mean between-visit interactions compared with White patients (0.46 [95% CI, 0.42-0.50] vs 0.59 [95% CI, 0.53-0.66] between-visit interactions/patient/mo; P < .001) and English-speaking patients (0.52 [95% CI, 0.47-0.58] vs 0.61 [95% CI, 0.56-0.66] between-visit interactions/patient/mo; P = .03).

Conclusions and Relevance

In this study, pre-PHE growth in overall patient-clinician encounters persisted after PHE-related telehealth implementation, driven in both periods by between-visit interactions. Differential utilization based on patient characteristics was observed, which may indicate disparities. The implications for health care team workload and patient outcomes are unknown, particularly regarding between-visit interactions. Therefore, to comprehensively understand care utilization for patients with chronic diseases, research should expand beyond billed visits.

Relief in Sight? Chatbots, In-baskets, and the Overwhelmed Primary Care Clinician

The recent emergence of publically facing artificial intelligence (AI) chatbots has generated vigorous discussion in the lay public around the possibilities, liabilities, and uncertainties of the integration of such technology into everyday life. As primary care clinicians continue to struggle against ever-increasing loads of asynchronous, electronic work, the potential for AI to improve the quality and efficiency of this work looms large. In this essay, we discuss the basic premise of open-access AI chatbots such as CHATGPT, review prior applications of AI in healthcare, and preview some possible AI chatbot-assisted in-basket assistance including scenarios of communicating test results with patients, providing patient education, and clinical decision support in history taking, review of prior diagnostic test characteristics, and common management scenarios. We discuss important concerns related to the future adoption of this technology including the transparency of the training data used in developing these models, the level of oversight and trustworthiness of the information generated, and possible impacts on equity, bias, and patient privacy. A stepwise and balanced approach to simultaneously understand the capabilities and address the concerns associated with these tools will be needed before these tools can improve patient care.

Oncology Clinicians' Perspectives on Online Patient Portal Use in Pediatric and Adolescent Cancer

PURPOSE

Online patient portals represent widely available communication tools in pediatric oncology. Previous studies have not evaluated clinicians' perspectives on portal use, including issues related to access to adolescents' portals.

METHODS

We performed semistructured interviews with physicians and advanced practice providers (APPs) who care for children or adolescents with cancer. We performed thematic analysis of benefits, problems, and accommodations related to portal use in oncology.

RESULTS

We interviewed 29 physicians and 24 APPs representing 26 institutions. Participants described five themes of benefits provided by portals: (1) empowering adolescents, (2) improving efficiency and accuracy of communication, (3) promoting open and adaptive communication, (4) supporting parents in managing care, and (5) bolstering clinical relationships. Participants described eight themes of problems caused by portal access: (1) creating emotional distress and confusion, (2) increasing workload and changing workflows, (3) threatening adolescent confidentiality, (4) adolescents lacking interest to engage, (5) diminishing clinical relationship, (6) misusing portal messages, (7) diminishing quality of sensitive documentation, and (8) parents losing access to adolescents' records. Participants described three themes related to accommodations they made as a result of portal access: (1) modifying note writing, (2) providing anticipatory guidance about viewing results, and (3) adapting workflows. Some portal functions created either benefits or problems depending on the clinical context.

CONCLUSION

Oncologists identified benefits and problems created by portal use, which were sometimes in tension, depending on the clinical context. To make portals useful, we must take steps to mitigate risks while preserving functionality for parents and adolescent patients.

Ctrl-C: a cross-sectional study of the electronic health record usage patterns of US oncology clinicians

Despite some positive impact, the use of electronic health records (EHRs) has been associated with negative effects, such as emotional exhaustion. We sought to compare EHR use patterns for oncology vs nononcology medical specialists. In this cross-sectional study, we employed EHR usage data for 349 ambulatory health-care systems nationwide collected from the vendor Epic from January to August 2019. We compared note composition, message volume, and time in the EHR system for oncology vs nononcology clinicians. Compared with nononcology medical specialists, oncologists had a statistically significantly greater percentage of notes derived from Copy and Paste functions but less SmartPhrase use. They received more total EHR messages per day than other medical specialists, with a higher proportion of results and system-generated messages. Our results point to priorities for enhancing EHR systems to meet the needs of oncology clinicians, particularly as related to facilitating the complex documentation, results, and therapy involved in oncology care.

Surgeon wellbeing in the 21st century

Physician time is under assault. Optimizing surgeons time and maximizing time spent on work that brings them the greatest professional fulfillment should be central tenants of these efforts.

The Role of Human and Organizational Factors in the Pursuit of One Digital Health

OBJECTIVE

This paper surveys a subset of the 2022 human and organizational factor (HOF) literature to provide guidance on building a One Digital Health ecosystem.

METHODS

We searched a subset of journals in PubMed/Medline for studies with "human factors" or "organization" in the title or abstract. Papers published in 2022 were eligible for inclusion in the survey. Selected papers were categorized into structural and behavioural aspects to understand digital health enabled interactions across micro, meso, and macro systems.

RESULTS

Our survey of the 2022 HOF literature showed that while we continue to make meaningful progress at digital health enabled interactions across systems levels, there are still challenges that must be overcome. For example, we must continue to grow the breadth of HOF research beyond individual users and systems to assist with the scale up of digital health systems across and beyond organizations. We summarize the findings by providing five HOF considerations to help build a One Digital Health ecosystem.

CONCLUSION

One Digital Health challenges us to improve coordination, communication, and collaboration between the health, environmental and veterinary sectors. Doing so requires us to develop both the structural and behavioural capacity of digital health systems at the organizational level and beyond so that we can develop more robust and integrated systems across health, environmental and veterinary sectors. The HOF community has much to offer and must play a leading role in designing a One Digital Health ecosystem.

Leveraging Large Language Models for Generating Responses to Patient Messages

Objective

This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal.

Methods

Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate the fine-tuned models, we used ten representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness.

Results

The dataset consisted of a total of 499,794 pairs of patient messages and corresponding responses from the patient portal, with 5,000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider's responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT's responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness.

Conclusion

Leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and primary care providers.

Putting ChatGPT's Medical Advice to the (Turing) Test: Survey Study

BACKGROUND

Chatbots are being piloted to draft responses to patient questions, but patients' ability to distinguish between provider and chatbot responses and patients' trust in chatbots' functions are not well established.

OBJECTIVE

This study aimed to assess the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar artificial intelligence-based chatbot for patient-provider communication.

METHODS

A survey study was conducted in January 2023. Ten representative, nonadministrative patient-provider interactions were extracted from the electronic health record. Patients' questions were entered into ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider's response. In the survey, each patient question was followed by a provider- or ChatGPT-generated response. Participants were informed that 5 responses were provider generated and 5 were chatbot generated. Participants were asked-and incentivized financially-to correctly identify the response source. Participants were also asked about their trust in chatbots' functions in patient-provider communication, using a Likert scale from 1-5.

RESULTS

A US-representative sample of 430 study participants aged 18 and older were recruited on Prolific, a crowdsourcing platform for academic studies. In all, 426 participants filled out the full survey. After removing participants who spent less than 3 minutes on the survey, 392 respondents remained. Overall, 53.3% (209/392) of respondents analyzed were women, and the average age was 47.1 (range 18-91) years. The correct classification of responses ranged between 49% (192/392) to 85.7% (336/392) for different questions. On average, chatbot responses were identified correctly in 65.5% (1284/1960) of the cases, and human provider responses were identified correctly in 65.1% (1276/1960) of the cases. On average, responses toward patients' trust in chatbots' functions were weakly positive (mean Likert score 3.4 out of 5), with lower trust as the health-related complexity of the task in the questions increased.

CONCLUSIONS

ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower-risk health questions. It is important to continue studying patient-chatbot interaction as chatbots move from administrative to more clinical roles in health care.

Experience and attitudes of pharmacists towards challenges and adaptive measures to new norm in ward pharmacy practice during the COVID-19 pandemic

BACKGROUND

COVID-19 pandemic has created challenges to the ward pharmacy practice. Challenges arose due to new norms in the ward pharmacy practice. Adaptive measures to overcome these challenges were important to sustain the quality of pharmaceutical care. This study aimed to identify the perceived challenges and attitudes towards adaptive measures in the ward pharmacy practice during the COVID-19 pandemic and determined their association with pharmacists' characteristics.

METHOD

This cross-sectional study was conducted in 14 Perak state hospitals and 12 primary health clinics through an online survey. All ward pharmacists and trainee pharmacists with at least 1 month of ward pharmacy experience and working in government-funded health facilities were included. The validated survey tool consisted of demographic characteristics, pharmacists' experience towards challenges (22 items), and their attitude towards adaptive measures (9 items). Each item was measured based on a 5-point Likert scale. One-way ANOVA and logistic regression were employed to determine the association of pharmacists' characteristics against their experience and attitude.

RESULTS

Out of 175 respondents, 144 (81.8%) were female, and 84 (47.7%) were Chinese. Most pharmacists served in the medical ward (124, 70.5%). Commonly reported perceived challenges were difficulties in counselling medication devices (3.63 ± 1.06), difficulties in clerking medication history from family members (3.63 ± 0.99), contacting family members (3.46 ± 0.90), patient's digital illiteracy in virtual counselling (3.43 ± 1.11) and completeness of the electronic records (3.36 ± 0.99). For attitude towards adaptive measures, improving internet connection (4.62 ± 0.58), ensuring availability of multilingual counselling videos (4.45 ± 0.64), and provision of internet-enabled mobile devices (4.39 ± 0.76) were the most agreeable by the pharmacists. Male (AOR: 2.63, CI 1.12-6.16, p = 0.026) and master's degree holders (AOR: 2.79, CI 0.95-8.25, p = 0.063) had greater odds of high perceived challenging experience scores. Master's degree holders (AOR: 8.56, CI 1.741-42.069, p = 0.008) were also more likely to have a positive attitude score towards adaptive measures.

CONCLUSION

Pharmacists faced multiple challenges in the ward pharmacy practice during the COVID-19 pandemic, especially in medication history assessment and patient counselling. Pharmacists, especially those with higher levels of education and longer tenure, exhibited a higher level of agreement towards the adaptive measures. The positive attitudes of pharmacists towards various adaptive measures, such as improvement of internet infrastructure and digital health literacy among patients and family members, warrant immediate action plans from health authorities.

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

Importance

The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians.

Objective

To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions.

Design, Setting, and Participants

In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians.

Results

Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P < .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot.

Conclusions

In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

Clinical trial knowledge among U.S. adults aged 65 years and up: Findings from a 2020 national survey

BACKGROUND

Older adults are underrepresented in most clinical trials. As the United States observes growth in the number of older adults over time, it will be critical to include them in clinical trials to improve the generalizability of results across age groups. Although clinical trial participation requires clinical trial knowledge, no study has assessed clinical trial knowledge among older adults. Using a national survey, this study aims to identify the prevalence and determinants of clinical trial knowledge among older adults.

METHODS

We used the 2020 Health Information National Trends Survey for secondary data analysis. We restricted the sample to older adults (aged 65 years and up). Our outcome variable was whether respondents reported having any clinical trial knowledge. We controlled for demographics, social determinants of health, healthcare utilization, and comorbidities through multivariable logistic regression models.

RESULTS

Using a weighted sample of 27,574,958 adults, we estimated that 61.1% of older adults reported having at least some knowledge of clinical trials. After controlling for other factors, those with one to two (OR = 1.80, 95% CI:1.14-2.84) or three to five (OR = 2.93, 95% CI:1.74-4.95) portal visits compared with no portal visits, those with cancer (OR = 1.92, 95% CI:1.22-3.02), and those with depression (OR = 2.27, 95% CI:1.23-4.20) had greater odds of having clinical trial knowledge. Inversely, those with hypertension (OR = 0.62, 95% CI:0.42-0.92) had lower odds of clinical trial knowledge.

CONCLUSIONS

Patient portal recruitment may be a supplemental intervention to improve clinical trial knowledge among older adults. Further research on additional interventions for identifying eligible participants is needed to minimize the burden among clinicians amidst other competing demands during clinic visits.

Using Time Series Clustering to Segment and Infer Emergency Department Nursing Shifts from Electronic Health Record Log Files

Few computational approaches exist for abstracting electronic health record (EHR) log files into clinically meaningful phenomena like clinician shifts. Because shifts are a fundamental unit of work recognized in clinical settings, shifts may serve as a primary unit of analysis in the study of documentation burden. We conducted a proof- of-concept study to investigate the feasibility of a novel approach using time series clustering to segment and infer clinician shifts from EHR log files. From 33,535,585 events captured between April-June 2021, we computationally identified 43,911 potential shifts among 2,285 (74.2%) emergency department nurses. On average, computationally-identified shifts were 10.6±3.1 hours long. Based on data distributions, we classified these shifts based on type: day, evening, night; and length: 12-hour, 8-hour, other. We validated our method through manual chart review of computationally-identified 12-hour shifts achieving 92.0% accuracy. Preliminary results suggest unsupervised clustering methods may be a reasonable approach for rapidly identifying clinician shifts.