Association Between Electronic Health Record Time and Quality of Care Metrics in Primary Care

Electronic health record and primary care physician self-reported quality of care: a multilevel study in China

BACKGROUND

Health information technology is one of the building blocks of a high-performing health system. However, the evidence regarding the influence of an electronic health record (EHR) on the quality of care remains mixed, especially in low- and middle-income countries.

OBJECTIVE

This study examines the association between greater EHR functionality and primary care physician self-reported quality of care.

METHODS

A total of 224 primary care physicians from 38 community health centres (CHCs) in four large Chinese cities participated in a cross-sectional survey to assess CHC care quality. Each CHC director scored their CHC's EHR functionality on the availability of ten typical features covering health information, data, results management, patient access, and clinical decision support. Data analysis utilised hierarchical linear modelling.

RESULTS

The availability of five EHR features was positively associated with physician self-reported clinical quality: share records online with providers outside the practice (β = 0.276, p = 0.04), access records online by the patient (β = 0.325, p = 0.04), alert provider of potential prescription problems (β = 0.353, p = 0.04), send the patient reminders for care (β = 0.419, p = 0.003), and list patients by diagnosis or health risk (β = 0.282, p = 0.04). However, no association was found between specific features availability or total features score and physician self-reported preventive quality.

CONCLUSIONS

This study provides evidence that the availability of EHR systems, and specific features of these systems, was positively associated with physician self-reported quality of care in these 38 CHCs. Future longitudinal studies focused on standardised quality metrics, and designed to control known confounding variables, will further inform quality improvement efforts in primary care.

Improving quality of care by standardising patient data collection in electronic medical records in an oncology department in Spain

BACKGROUND

Evaluation of quality of care in oncology is key in ensuring patients receive adequate treatment. American Society of Clinical Oncology's (ASCO) Quality Oncology Practice Initiative (QOPI) Certification Program (QCP) is an international initiative that evaluates quality of care in outpatient oncology practices.

METHODS

We retrospectively reviewed free-text electronic medical records from patients with breast cancer (BR), colorectal cancer (CRC) or non-small cell lung cancer (NSCLC). In a baseline measurement, high scores were obtained for the nine disease-specific measures of QCP Track (2021 version had 26 measures); thus, they were not further analysed. We evaluated two sets of measures: the remaining 17 QCP Track measures, as well as these plus other 17 measures selected by us (combined measures). Review of data from 58 patients (26 BR; 18 CRC; 14 NSCLC) seen in June 2021 revealed low overall quality scores (OQS)-below ASCO's 75% threshold-for QCP Track measures (46%) and combined measures (58%). We developed a plan to improve OQS and monitored the impact of the intervention by abstracting data at subsequent time points.

RESULTS

We evaluated potential causes for the low OQS and developed a plan to improve it over time by educating oncologists at our hospital on the importance of improving collection of measures and highlighting the goal of applying for QOPI certification. We conducted seven plan-do-study-act cycles and evaluated the scores at seven subsequent data abstraction time points from November 2021 to December 2022, reviewing 404 patients (199 BR; 114 CRC; 91 NSCLC). All measures were improved. Four months after the intervention, OQS surpassed the quality threshold and was maintained for 10 months until the end of the study (range, 78-87% for QCP Track measures; 78-86% for combined measures).

CONCLUSIONS

We developed an easy-to-implement intervention that achieved a fast improvement in OQS, enabling our Medical Oncology Department to aim for QOPI certification.

Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis

OBJECTIVE

This study aims to evaluate the accuracy of ChatGPT in answering allergic rhinitis (AR) and chronic rhinosinusitis (CRS) related questions.

STUDY DESIGN

This is a cross-sectional study.

SETTING

Each question was inputted as a separate, independent prompt.

METHODS

Responses to AR (n = 189) and CRS (n = 242) related questions, generated by GPT-3.5 and GPT-4, were independently graded for accuracy by 2 senior rhinology professors, with disagreements adjudicated by a third reviewer.

RESULTS

Overall, ChatGPT demonstrated a satisfactory performance, accurately answering over 80% of questions across all categories. Specifically, GPT-4.0's accuracy in responding to AR-related questions significantly exceeded that of GPT-3.5, but distinction not evident in CRS-related questions. Patient-originated questions had a significantly higher accuracy compared to doctor-originated questions when utilizing GPT-4.0 to respond to AR-related questions. This discrepancy was not observed with GPT-3.5 or in the context of CRS-related questions. Across different types of content, ChatGPT excelled in covering basic knowledge, prevention, and emotion for AR and CRS. However, it experienced challenges when addressing questions about recent advancements, a trend consistent across both GPT-3.5 and GPT-4.0 iterations. Importantly, the accuracy of responses remained unaffected when questions were posed in Chinese.

CONCLUSION

Our findings suggest ChatGPT's capability to convey accurate information for AR and CRS patients, and offer insights into its performance across various domains, guiding its utilization and improvement.

Primary Care Physician Preferences Regarding Communication from Orthopaedic Surgeons

BACKGROUND

Musculoskeletal consultations constitute a growing portion of primary care physician (PCP) referrals. Optimizing communication between PCPs and orthopaedists can potentially reduce time spent in the electronic medical record (EMR) as well as physician burnout. Little is known about the preferences of PCPs regarding communication from orthopaedic surgeons. Hence, the present study investigated, across a large health network, the preferences of PCPs regarding communication from orthopaedists.

METHODS

A total of 175 PCPs across 15 practices within our health network were surveyed. These providers universally utilized Epic as their EMR platform. Five-point, labeled Likert scales were utilized to assess the PCP-perceived importance of communication from orthopaedists in specific clinical scenarios. PCPs were further asked to report their preferred method of communication in each scenario and their overall interest in communication from orthopaedists. Logistic regression analyses were performed to determine whether any PCP characteristics were associated with the preferred method of communication and the overall PCP interest in communication from orthopaedists.

RESULTS

A total of 107 PCPs (61.1%) responded to the survey. PCPs most commonly rated communication from orthopaedists as highly important in the scenario of an orthopaedist needing information from the PCP. In this scenario, PCPs preferred to receive an Epic Staff Message. Scenarios involving a recommendation for surgery, hospitalization, or a major clinical change were also rated as highly important. In these scenarios, an Epic CC'd Chart rather than a Staff Message was preferred. Increased after-hours EMR use was associated with diminished odds of having a high interest in communication from orthopaedists (odds ratio, 0.65; 95% confidence interval, 0.48 to 0.88; p = 0.005). Ninety-three PCPs (86.9%) reported spending 1 to 1.5 hours or more per day in Epic after normal clinical hours, and 27 (25.2%) spent >3 hours per day. Forty-six PCPs (43.0%) reported experiencing ≥1 symptom of burnout.

CONCLUSIONS

There were distinct preferences among PCPs regarding clinical communication from orthopaedic surgeons. There was also evidence of substantial burnout and after-hours work effort by PCPs. These results may help to optimize communication between PCPs and orthopaedists while reducing the amount of time that PCPs spend in the EMR.

Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support

INTRODUCTION

Sialendoscopy has emerged in the last decades as a groundbreaking technique, offering a minimally invasive approach for exploring and managing salivary gland disorders. More recently, the advent of chatbots, powered by advanced natural processing language and artificial intelligence algorithms, has revolutionized the way healthcare professionals and patients access and analyze medical information and potentially will support soon the clinical decision-making process.

MATERIALS AND METHODS

A prospective, cross-sectional study was designed to assess the level of agreement between Chat-GPT and 10 expert sialendoscopists aiming the capabilities of Chat-GPT to further improve the management of salivary gland disorders.

RESULTS

The mean level of agreement was 3.4 (SD: 0.69; Min: 2, Max: 4) for Chat-GPT's answers while it was 4.1 (SD: 0.56; Min: 3, Max: 5) for the group of EESS (p < 0.015). The overall Wilcoxon signed-rank test yielded a significance level of p < 0.026 when comparing the level of agreement between Chat-GPT and EESS. The mean number of therapeutic alternatives suggested by Chat-GPT was 3.33 (SD: 1.2; Min: 2, Max: 5), while it was 2.6 (SD: 0.51; Min: 2, Max: 3) for the group of EESS; p = 0.286 (95% CI - 0.385 to 1.320).

CONCLUSION

Chat-GPT represents a promising tool in the clinical decision-making process within the salivary gland clinic, particularly for patients who are candidates for sialendoscopy treatment. Additionally, it serves as a valuable source of information for patients. However, further development is necessary to enhance the reliability of these tools and ensure their safety and optimal use in the clinical setting.

Improved accuracy and efficiency of primary care fall risk screening of older adults using a machine learning approach

BACKGROUND

While many falls are preventable, they remain a leading cause of injury and death in older adults. Primary care clinics largely rely on screening questionnaires to identify people at risk of falls. Limitations of standard fall risk screening questionnaires include suboptimal accuracy, missing data, and non-standard formats, which hinder early identification of risk and prevention of fall injury. We used machine learning methods to develop and evaluate electronic health record (EHR)-based tools to identify older adults at risk of fall-related injuries in a primary care population and compared this approach to standard fall screening questionnaires.

METHODS

Using patient-level clinical data from an integrated healthcare system consisting of 16-member institutions, we conducted a case-control study to develop and evaluate prediction models for fall-related injuries in older adults. Questionnaire-derived prediction with three questions from a commonly used fall risk screening tool was evaluated. We then developed four temporal machine learning models using routinely available longitudinal EHR data to predict the future risk of fall injury. We also developed a fall injury-prevention clinical decision support (CDS) implementation prototype to link preventative interventions to patient-specific fall injury risk factors.

RESULTS

Questionnaire-based risk screening achieved area under the receiver operating characteristic curve (AUC) up to 0.59 with 23% to 33% similarity for each pair of three fall injury screening questions. EHR-based machine learning risk screening showed significantly improved performance (best AUROC = 0.76), with similar prediction performance between 6-month and one-year prediction models.

CONCLUSIONS

The current method of questionnaire-based fall risk screening of older adults is suboptimal with redundant items, inadequate precision, and no linkage to prevention. A machine learning fall injury prediction method can accurately predict risk with superior sensitivity while freeing up clinical time for initiating personalized fall prevention interventions. The developed algorithm and data science pipeline can impact routine primary care fall prevention practice.

Race, ethnicity, ancestry, and aspects that impact HLA data and matching for transplant

Race, ethnicity, and ancestry are terms that are often misinterpreted and/or used interchangeably. There is lack of consensus in the scientific literature on the definition of these terms and insufficient guidelines on the proper classification, collection, and application of this data in the scientific community. However, defining groups for human populations is crucial for multiple healthcare applications and clinical research. Some examples impacted by population classification include HLA matching for stem-cell or solid organ transplant, identifying disease associations and/or adverse drug reactions, defining social determinants of health, understanding diverse representation in research studies, and identifying potential biases. This article describes aspects of race, ethnicity and ancestry information that impact the stem-cell or solid organ transplantation field with particular focus on HLA data collected from donors and recipients by donor registries or transplant centers.

Guidance for reporting analyses of metadata on electronic health record use

INTRODUCTION

Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies.

RECOMMENDATIONS

In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

The role of organizations in shaping physician use of electronic health records

OBJECTIVE

The aim of the study was to (1) characterize organizational differences in primary care physicians' electronic health record (EHR) behavior; (2) assess within-organization consistency in EHR behaviors; and (3) identify whether organizational consistency is associated with physician-level efficiency.

DATA SOURCES

EHR metadata capturing averaged weekly measures of EHR time and documentation composition from 75,124 US primary care physicians across 299 organizations between September 2020 and May 2021 were taken. EHR time measures include active time in orders, chart review, notes, messaging, time spent outside of scheduled hours, and total EHR time. Documentation composition measures include note length and percentage use of templated text or copy/paste. Efficiency is measured as the percent of visits with same-day note completion.

STUDY DESIGN

All analyses are cross-sectional. Across-organization differences in EHR use and documentation composition are presented via 90th-to-10th percentile ratios of means and SDs. Multilevel modeling with post-estimation variance partitioning assesses the extent of an organizational signature-the proportion of variation in our measures attributable to organizations (versus specialty and individual behaviors). We measured organizational internal consistency for each measure via organization-level SD, which we grouped into quartiles for regression. Association between internally consistent (i.e., low SD) organizational EHR use and physician-level efficiency was assessed with multi-variable OLS models.

DATA COLLECTION

Extraction from Epic's Signal platform used for measuring provider EHR efficiency.

PRINCIPAL FINDINGS

EHR time per visit for physicians at a 90th percentile organization is 1.94 times the average EHR time at a 10th percentile organization. There is little evidence, on average, of an organizational signature. However, physicians in organizations with high internal consistency in EHR use demonstrate increased efficiency. Physicians in organizations with the highest internal consistency (top quartile) have a 3.77 percentage point higher same-day visit closure rates compared with peers in bottom quartile organizations (95% confidence interval: 0.0142-0.0612).

CONCLUSIONS

Results suggest unrealized opportunities for organizations and policymakers to support consistency in how physicians engage in EHR-supported work.

Navigating the doctor-patient-AI relationship - a mixed-methods study of physician attitudes toward artificial intelligence in primary care

BACKGROUND

Artificial intelligence (AI) is a rapidly advancing field that is beginning to enter the practice of medicine. Primary care is a cornerstone of medicine and deals with challenges such as physician shortage and burnout which impact patient care. AI and its application via digital health is increasingly presented as a possible solution. However, there is a scarcity of research focusing on primary care physician (PCP) attitudes toward AI. This study examines PCP views on AI in primary care. We explore its potential impact on topics pertinent to primary care such as the doctor-patient relationship and clinical workflow. By doing so, we aim to inform primary care stakeholders to encourage successful, equitable uptake of future AI tools. Our study is the first to our knowledge to explore PCP attitudes using specific primary care AI use cases rather than discussing AI in medicine in general terms.

METHODS

From June to August 2023, we conducted a survey among 47 primary care physicians affiliated with a large academic health system in Southern California. The survey quantified attitudes toward AI in general as well as concerning two specific AI use cases. Additionally, we conducted interviews with 15 survey respondents.

RESULTS

Our findings suggest that PCPs have largely positive views of AI. However, attitudes often hinged on the context of adoption. While some concerns reported by PCPs regarding AI in primary care focused on technology (accuracy, safety, bias), many focused on people-and-process factors (workflow, equity, reimbursement, doctor-patient relationship).

CONCLUSION

Our study offers nuanced insights into PCP attitudes towards AI in primary care and highlights the need for primary care stakeholder alignment on key issues raised by PCPs. AI initiatives that fail to address both the technological and people-and-process concerns raised by PCPs may struggle to make an impact.

System-Level Factors and Time Spent on Electronic Health Records by Primary Care Physicians

Importance

Primary care physicians (PCPs) spend the most time on the electronic health record (EHR) of any specialty. Thus, it is critical to understand what factors contribute to varying levels of PCP time spent on EHRs.

Objective

To characterize variation in EHR time across PCPs and primary care clinics, and to describe how specific PCP, patient panel, clinic, and team collaboration factors are associated with PCPs' time spent on EHRs.

Design, Setting, and Participants

This cross-sectional study included 307 PCPs practicing across 31 primary care clinics at Massachusetts General Hospital and Brigham and Women's Hospital during 2021. Data were analyzed from October 2022 to October 2023.

Main Outcomes and Measures

Total per-visit EHR time, total per-visit pajama time (ie, time spent on the EHR between 5:30 pm to 7:00 am and on weekends), and total per-visit time on the electronic inbox as measured by activity log data derived from an EHR database.

Results

The sample included 307 PCPs (183 [59.6%] female). On a per-visit basis, PCPs spent a median (IQR) of 36.2 (28.9-45.7) total minutes on the EHR, 6.2 (3.1-11.5) minutes of pajama time, and 7.8 (5.5-10.7) minutes on the electronic inbox. When comparing PCP time expenditure by clinic, median (IQR) total EHR time, median (IQR) pajama time, and median (IQR) electronic inbox time ranged from 23.5 (20.7-53.1) to 47.9 (30.6-70.7) minutes per visit, 1.7 (0.7-10.5) to 13.1 (7.7-28.2) minutes per visit, and 4.7 (4.1-5.2) to 10.8 (8.9-15.2) minutes per visit, respectively. In a multivariable model with an outcome of total per-visit EHR time per visit, an above median percentage of teamwork on orders was associated with 3.81 (95% CI, 0.49-7.13) minutes per visit fewer and having a clinic pharmacy technician was associated with 7.87 (95% CI, 2.03-13.72) minutes per visit fewer. Practicing in a community health center was associated with fewer minutes of total EHR time per visit (5.40 [95% CI, 0.06-10.74] minutes).

Conclusions and Relevance

There is substantial variation in EHR time among individual PCPs and PCPs within clinics. Organization-level factors, such as team collaboration on orders, support for medication refill functions, and practicing in a community health center, are associated with lower EHR time for PCPs. These findings highlight the importance of addressing EHR burden at a systems level.

Differences in Health Professionals' Engagement With Electronic Health Records Based on Inpatient Race and Ethnicity

Importance

US health professionals devote a large amount of effort to engaging with patients' electronic health records (EHRs) to deliver care. It is unknown whether patients with different racial and ethnic backgrounds receive equal EHR engagement.

Objective

To investigate whether there are differences in the level of health professionals' EHR engagement for hospitalized patients according to race or ethnicity during inpatient care.

Design, Setting, and Participants

This cross-sectional study analyzed EHR access log data from 2 major medical institutions, Vanderbilt University Medical Center (VUMC) and Northwestern Medicine (NW Medicine), over a 3-year period from January 1, 2018, to December 31, 2020. The study included all adult patients (aged ≥18 years) who were discharged alive after hospitalization for at least 24 hours. The data were analyzed between August 15, 2022, and March 15, 2023.

Exposures

The actions of health professionals in each patient's EHR were based on EHR access log data. Covariates included patients' demographic information, socioeconomic characteristics, and comorbidities.

Main Outcomes and Measures

The primary outcome was the quantity of EHR engagement, as defined by the average number of EHR actions performed by health professionals within a patient's EHR per hour during the patient's hospital stay. Proportional odds logistic regression was applied based on outcome quartiles.

Results

A total of 243 416 adult patients were included from VUMC (mean [SD] age, 51.7 [19.2] years; 54.9% female and 45.1% male; 14.8% Black, 4.9% Hispanic, 77.7% White, and 2.6% other races and ethnicities) and NW Medicine (mean [SD] age, 52.8 [20.6] years; 65.2% female and 34.8% male; 11.7% Black, 12.1% Hispanic, 69.2% White, and 7.0% other races and ethnicities). When combining Black, Hispanic, or other race and ethnicity patients into 1 group, these patients were significantly less likely to receive a higher amount of EHR engagement compared with White patients (adjusted odds ratios, 0.86 [95% CI, 0.83-0.88; P < .001] for VUMC and 0.90 [95% CI, 0.88-0.92; P < .001] for NW Medicine). However, a reduction in this difference was observed from 2018 to 2020.

Conclusions and Relevance

In this cross-sectional study of inpatient EHR engagement, the findings highlight differences in how health professionals distribute their efforts to patients' EHRs, as well as a method to measure these differences. Further investigations are needed to determine whether and how EHR engagement differences are correlated with health care outcomes.

COVID exacerbated the gender disparity in physician electronic health record inbox burden

The COVID-19 pandemic was associated with significant changes to the delivery of ambulatory care, including a dramatic increase in patient messages to physicians. While asynchronous messaging is a valuable communication modality for patients, a greater volume of patient messages is associated with burnout and decreased well-being for physicians. Given that women physicians experienced greater electronic health record (EHR) burden and received more patient messages pre-pandemic, there is concern that COVID may have exacerbated this disparity. Using EHR audit log data of ambulatory physicians at an academic medical center, we used a difference-in-differences framework to evaluate the impact of the pandemic on patient message volume and compare differences between men and women physicians. We found patient message volume increased post-COVID for all physicians, and women physicians saw an additional increase compared to men. Our results contribute to the growing evidence of different communication expectations for women physicians that contribute to the gender disparity in EHR burden.

Use of Natural Language Processing of Patient-Initiated Electronic Health Record Messages to Identify Patients With COVID-19 Infection

Importance

Natural language processing (NLP) has the potential to enable faster treatment access by reducing clinician response time and improving electronic health record (EHR) efficiency.

Objective

To develop an NLP model that can accurately classify patient-initiated EHR messages and triage COVID-19 cases to reduce clinician response time and improve access to antiviral treatment.

Design, Setting, and Participants

This retrospective cohort study assessed development of a novel NLP framework to classify patient-initiated EHR messages and subsequently evaluate the model's accuracy. Included patients sent messages via the EHR patient portal from 5 Atlanta, Georgia, hospitals between March 30 and September 1, 2022. Assessment of the model's accuracy consisted of manual review of message contents to confirm the classification label by a team of physicians, nurses, and medical students, followed by retrospective propensity score-matched clinical outcomes analysis.

Exposure

Prescription of antiviral treatment for COVID-19.

Main Outcomes and Measures

The 2 primary outcomes were (1) physician-validated evaluation of the NLP model's message classification accuracy and (2) analysis of the model's potential clinical effect via increased patient access to treatment. The model classified messages into COVID-19-other (pertaining to COVID-19 but not reporting a positive test), COVID-19-positive (reporting a positive at-home COVID-19 test result), and non-COVID-19 (not pertaining to COVID-19).

Results

Among 10 172 patients whose messages were included in analyses, the mean (SD) age was 58 (17) years; 6509 patients (64.0%) were women and 3663 (36.0%) were men. In terms of race and ethnicity, 2544 patients (25.0%) were African American or Black, 20 (0.2%) were American Indian or Alaska Native, 1508 (14.8%) were Asian, 28 (0.3%) were Native Hawaiian or other Pacific Islander, 5980 (58.8%) were White, 91 (0.9%) were more than 1 race or ethnicity, and 1 (0.01%) chose not to answer. The NLP model had high accuracy and sensitivity, with a macro F1 score of 94% and sensitivity of 85% for COVID-19-other, 96% for COVID-19-positive, and 100% for non-COVID-19 messages. Among the 3048 patient-generated messages reporting positive SARS-CoV-2 test results, 2982 (97.8%) were not documented in structured EHR data. Mean (SD) message response time for COVID-19-positive patients who received treatment (364.10 [784.47] minutes) was faster than for those who did not (490.38 [1132.14] minutes; P = .03). Likelihood of antiviral prescription was inversely correlated with message response time (odds ratio, 0.99 [95% CI, 0.98-1.00]; P = .003).

Conclusions and Relevance

In this cohort study of 2982 COVID-19-positive patients, a novel NLP model classified patient-initiated EHR messages reporting positive COVID-19 test results with high sensitivity. Furthermore, when responses to patient messages occurred faster, patients were more likely to receive antiviral medical prescription within the 5-day treatment window. Although additional analysis on the effect on clinical outcomes is needed, these findings represent a possible use case for integration of NLP algorithms into clinical care.

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

Importance

The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians.

Objective

To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions.

Design, Setting, and Participants

In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians.

Results

Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P < .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot.

Conclusions

In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

How does work environment relate to diagnostic quality? A prospective, mixed methods study in primary care

OBJECTIVES

The quest to measure and improve diagnosis has proven challenging; new approaches are needed to better understand and measure key elements of the diagnostic process in clinical encounters. The aim of this study was to develop a tool assessing key elements of the diagnostic assessment process and apply it to a series of diagnostic encounters examining clinical notes and encounters' recorded transcripts. Additionally, we aimed to correlate and contextualise these findings with measures of encounter time and physician burnout.

DESIGN

We audio-recorded encounters, reviewed their transcripts and associated them with their clinical notes and findings were correlated with concurrent Mini Z Worklife measures and physician burnout.

SETTING

Three primary urgent-care settings.

PARTICIPANTS

We conducted in-depth evaluations of 28 clinical encounters delivered by seven physicians.

RESULTS

Comparing encounter transcripts with clinical notes, in 24 of 28 (86%) there was high note/transcript concordance for the diagnostic elements on our tool. Reliably included elements were red flags (92% of notes/encounters), aetiologies (88%), likelihood/uncertainties (71%) and follow-up contingencies (71%), whereas psychosocial/contextual information (35%) and mentioning common pitfalls (7%) were often missing. In 22% of encounters, follow-up contingencies were in the note, but absent from the recorded encounter. There was a trend for higher burnout scores being associated with physicians less likely to address key diagnosis items, such as psychosocial history/context.

CONCLUSIONS

A new tool shows promise as a means of assessing key elements of diagnostic quality in clinical encounters. Work conditions and physician reactions appear to correlate with diagnostic behaviours. Future research should continue to assess relationships between time pressure and diagnostic quality.