Barriers and enablers for externally and internally driven implementation processes in healthcare: a qualitative cross-case study

BACKGROUND

Quality in healthcare is a subject in need of continuous attention. Quality improvement (QI) programmes with the purpose of increasing service quality are therefore of priority for healthcare leaders and governments. This study explores the implementation process of two different QI programmes, one externally driven implementation and one internally driven, in Norwegian nursing homes and home care services. The aim for the study was to identify enablers and barriers for externally and internally driven implementation processes in nursing homes and homecare services, and furthermore to explore if identified enablers and barriers are different or similar across the different implementation processes.

METHODS

This study is based on an exploratory qualitative methodology. The empirical data was collected through the 'Improving Quality and Safety in Primary Care - Implementing a Leadership Intervention in Nursing Homes and Homecare' (SAFE-LEAD) project. The SAFE-LEAD project is a multiple case study of two different QI programmes in primary care in Norway. A large externally driven implementation process was supplemented with a tracer project involving an internally driven implementation process to identify differences and similarities. The empirical data was inductively analysed in accordance with grounded theory.

RESULTS

Enablers for both external and internal implementation processes were found to be technology and tools, dedication, and ownership. Other more implementation process specific enablers entailed continuous learning, simulation training, knowledge sharing, perceived relevance, dedication, ownership, technology and tools, a systematic approach and coordination. Only workload was identified as coincident barriers across both externally and internally implementation processes. Implementation process specific barriers included turnover, coping with given responsibilities, staff variety, challenges in coordination, technology and tools, standardizations not aligned with work, extensive documentation, lack of knowledge sharing.

CONCLUSION

This study provides understanding that some enablers and barriers are present in both externally and internally driven implementation processes, while other are more implementation process specific. Dedication, engagement, technology and tools are coinciding enablers which can be drawn upon in different implementation processes, while workload acted as the main barrier in both externally and internally driven implementation processes. This means that some enablers and barriers can be expected in implementation of QI programmes in nursing homes and home care services, while others require contextual understanding of their setting and work.

Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool

OBJECTIVE

To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high one-year mortality risk.

MATERIALS AND METHODS

We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low-risk of one-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality.

RESULTS

Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (SD) age of 69.3 (17.5) years, and a mean (SD) Charlson comorbidity index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high-risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high-risk (68.7% concordance). For 105 patients whom PCPs deemed SIC-unnecessary, the tool classified 83 as low-risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640).

CONCLUSIONS AND RELEVANCE

A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

A qualitative study of leaders' experiences of handling challenges and changes induced by the COVID-19 pandemic in rural nursing homes and homecare services

BACKGROUND

The COVID-19 pandemic had a major impact on healthcare services globally. In care settings such as small rural nursing homes and homes care services leaders were forced to confront, and adapt to, both new and ongoing challenges to protect their employees and patients and maintain their organization's operation. The aim of this study was to assess how healthcare leaders, working in rural primary healthcare services, led nursing homes and homecare services during the COVID-19 pandemic. Moreover, the study sought to explore how adaptations to changes and challenges induced by the pandemic were handled by leaders in rural nursing homes and homecare services.

METHODS

The study employed a qualitative explorative design with individual interviews. Nine leaders at different levels, working in small, rural nursing homes and homecare services in western Norway were included.

RESULTS

Three main themes emerged from the thematic analysis: "Navigating the role of a leader during the pandemic," "The aftermath - management of COVID-19 in rural primary healthcare services", and "The benefits and drawbacks of being small and rural during the pandemic."

CONCLUSIONS

Leaders in rural nursing homes and homecare services handled a multitude of immediate challenges and used a variety of adaptive strategies during the COVID-19 pandemic. While handling their own uncertainty and rapidly changing roles, they also coped with organizational challenges and adopted strategies to maintain good working conditions for their employees, as well as maintain sound healthcare management. The study results establish the intricate nature of resilient leadership, encompassing individual resilience, personality, governance, resource availability, and the capability to adjust to organizational and employee requirements, and how the rural context may affect these aspects.

The potential for artificial intelligence to transform healthcare: perspectives from international health leaders

Artificial intelligence (AI) has the potential to transform care delivery by improving health outcomes, patient safety, and the affordability and accessibility of high-quality care. AI will be critical to building an infrastructure capable of caring for an increasingly aging population, utilizing an ever-increasing knowledge of disease and options for precision treatments, and combatting workforce shortages and burnout of medical professionals. However, we are not currently on track to create this future. This is in part because the health data needed to train, test, use, and surveil these tools are generally neither standardized nor accessible. There is also universal concern about the ability to monitor health AI tools for changes in performance as they are implemented in new places, used with diverse populations, and over time as health data may change. The Future of Health (FOH), an international community of senior health care leaders, collaborated with the Duke-Margolis Institute for Health Policy to conduct a literature review, expert convening, and consensus-building exercise around this topic. This commentary summarizes the four priority action areas and recommendations for health care organizations and policymakers across the globe that FOH members identified as important for fully realizing AI's potential in health care: improving data quality to power AI, building infrastructure to encourage efficient and trustworthy development and evaluations, sharing data for better AI, and providing incentives to accelerate the progress and impact of AI.

Serum urate change among gout patients treated with sodium-glucose cotransporter type 2 inhibitors vs. sulfonylurea: A comparative effectiveness analysis

OBJECTIVE

To investigate the serum urate (SU) change among gout patients initiating SGLT2i, and to compare with sulfonylurea, the second-most widely used glucose-lowering medication after metformin.

METHODS

We conducted a cohort study of patients with gout and baseline SU >6 mg/dL who had SU measured within 90 days before and after SGLT2i or sulfonylurea initiation. Using multivariable linear regression, we compared SU change among SGLT2i initiators between those with and without diabetes and then compared SU change between SGLT2i and sulfonylurea.

RESULTS

We identified 28 patients with gout initiating SGLT2i (including 16 with diabetes) and 28 patients initiating sulfonylurea (all with diabetes). Among SGLT2i initiators, the mean within-group SU change was -1.8 (95 % CI, -2.4 to -1.1) mg/dL, including -1.2 (-1.8 to -0.6) mg/dL and -2.5 (-3.6 to -1.3) mg/dL among patients with and without diabetes, respectively, with an adjusted difference between those with and without diabetes of -1.4 (-2.4 to -0.5) mg/dL. The SU did not change after initiating sulfonylurea (+0.3 [-0.3 to 1.0] mg/dL). The adjusted SU change difference between SGLT2i vs. sulfonylurea initiation was -1.8 (-2.7 to -0.9) mg/dL in all patients. The SU reduction persisted regardless of urate-lowering therapy or diuretic use and the presence of diabetes, chronic kidney disease, or heart failure.

CONCLUSION

Among patients with gout, SGLT2i was associated with a notable reduction in SU compared with sulfonylurea, with a larger reduction among patients without diabetes. With their proven cardiovascular-kidney-metabolic benefits, adding SGLT2i to current gout management could provide streamlined benefits for gout and its comorbidities.

Utilization of electronic health record sex and gender demographic fields: a metadata and mixed methods analysis

OBJECTIVES

Despite federally mandated collection of sex and gender demographics in the electronic health record (EHR), longitudinal assessments are lacking. We assessed sex and gender demographic field utilization using EHR metadata.

MATERIALS AND METHODS

Patients ≥18 years of age in the Mass General Brigham health system with a first Legal Sex entry (registration requirement) between January 8, 2018 and January 1, 2022 were included in this retrospective study. Metadata for all sex and gender fields (Legal Sex, Sex Assigned at Birth [SAAB], Gender Identity) were quantified by completion rates, user types, and longitudinal change. A nested qualitative study of providers from specialties with high and low field use identified themes related to utilization.

RESULTS

1 576 120 patients met inclusion criteria: 100% had a Legal Sex, 20% a Gender Identity, and 19% a SAAB; 321 185 patients had field changes other than initial Legal Sex entry. About 2% of patients had a subsequent Legal Sex change, and 25% of those had ≥2 changes; 20% of patients had ≥1 update to Gender Identity and 19% to SAAB. Excluding the first Legal Sex entry, administrators made most changes (67%) across all fields, followed by patients (25%), providers (7.2%), and automated Health Level-7 (HL7) interface messages (0.7%). Provider utilization varied by subspecialty; themes related to systems barriers and personal perceptions were identified.

DISCUSSION

Sex and gender demographic fields are primarily used by administrators and raise concern about data accuracy; provider use is heterogenous and lacking. Provider awareness of field availability and variable workflows may impede use.

CONCLUSION

EHR metadata highlights areas for improvement of sex and gender field utilization.

Patient-reported outcome measures for primary hyperparathyroidism: a systematic review of measurement properties

BACKGROUND

The quality of patient-reported outcome measures (PROMs) used to assess the outcomes of primary hyperparathyroidism (PHPT), a common endocrine disorder that can negatively affect patients' health-related quality of life due to chronic symptoms, has not been rigorously examined. This systematic review aimed to summarize and evaluate evidence on the measurement properties of PROMs used in adult patients with PHPT, and to provide recommendations for appropriate measure selection.

METHODS

After PROSPERO registration (CRD42023438287), Medline, EMBASE, CINAHL Complete, Web of Science, PsycINFO, and Cochrane Trials were searched for full-text articles in English investigating PROM development, pilot studies, or evaluation of at least one PROM measurement property in adult patients with any clinical form of PHPT. Two reviewers independently identified studies for inclusion and conducted the review following the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) Methodology to assess risk of bias, evaluate the quality of measurement properties, and grade the certainty of evidence.

RESULTS

From 4989 records, nine PROM development or validation studies were identified for three PROMs: the SF-36, PAS, and PHPQoL. Though the PAS demonstrated sufficient test-retest reliability and convergent validity, and the PHPQoL sufficient test-retest reliability, convergent validity, and responsiveness, the certainty of evidence was low-to-very low due to risk of bias. All three PROMs lacked sufficient evidence for content validity in patients with PHPT.

CONCLUSIONS

Based upon the available evidence, the SF-36, PAS, and PHPQoL cannot currently be recommended for use in research or clinical care, raising important questions about the conclusions of studies using these PROMs. Further validation studies or the development of more relevant PROMs with strong measurement properties for this patient population are needed.

Using Artificial Intelligence to Improve Primary Care for Patients and Clinicians

This Viewpoint discusses how artificial intelligence can be used to increase efficiency of primary care processes for clinicians and patients.

Improved accuracy and efficiency of primary care fall risk screening of older adults using a machine learning approach

BACKGROUND

While many falls are preventable, they remain a leading cause of injury and death in older adults. Primary care clinics largely rely on screening questionnaires to identify people at risk of falls. Limitations of standard fall risk screening questionnaires include suboptimal accuracy, missing data, and non-standard formats, which hinder early identification of risk and prevention of fall injury. We used machine learning methods to develop and evaluate electronic health record (EHR)-based tools to identify older adults at risk of fall-related injuries in a primary care population and compared this approach to standard fall screening questionnaires.

METHODS

Using patient-level clinical data from an integrated healthcare system consisting of 16-member institutions, we conducted a case-control study to develop and evaluate prediction models for fall-related injuries in older adults. Questionnaire-derived prediction with three questions from a commonly used fall risk screening tool was evaluated. We then developed four temporal machine learning models using routinely available longitudinal EHR data to predict the future risk of fall injury. We also developed a fall injury-prevention clinical decision support (CDS) implementation prototype to link preventative interventions to patient-specific fall injury risk factors.

RESULTS

Questionnaire-based risk screening achieved area under the receiver operating characteristic curve (AUC) up to 0.59 with 23% to 33% similarity for each pair of three fall injury screening questions. EHR-based machine learning risk screening showed significantly improved performance (best AUROC = 0.76), with similar prediction performance between 6-month and one-year prediction models.

CONCLUSIONS

The current method of questionnaire-based fall risk screening of older adults is suboptimal with redundant items, inadequate precision, and no linkage to prevention. A machine learning fall injury prediction method can accurately predict risk with superior sensitivity while freeing up clinical time for initiating personalized fall prevention interventions. The developed algorithm and data science pipeline can impact routine primary care fall prevention practice.

Leveraging large language models to foster equity in healthcare

OBJECTIVES

Large language models (LLMs) are poised to change care delivery, but their impact on health equity is unclear. While marginalized populations have been historically excluded from early technology developments, LLMs present an opportunity to change our approach to developing, evaluating, and implementing new technologies. In this perspective, we describe the role of LLMs in supporting health equity.

MATERIALS AND METHODS

We apply the National Institute on Minority Health and Health Disparities (NIMHD) research framework to explore the use of LLMs for health equity.

RESULTS

We present opportunities for how LLMs can improve health equity across individual, family and organizational, community, and population health. We describe emerging concerns including biased data, limited technology diffusion, and privacy. Finally, we highlight recommendations focused on prompt engineering, retrieval augmentation, digital inclusion, transparency, and bias mitigation.

CONCLUSION

The potential of LLMs to support health equity depends on making health equity a focus from the start.

Patient Safety Indicators During the Initial COVID-19 Pandemic Surge in the United States

OBJECTIVE

The COVID-19 pandemic presented a challenge to inpatient safety. It is unknown whether there were spillover effects due to COVID-19 into non-COVID-19 care and safety. We sought to evaluate the changes in inpatient Agency for Healthcare Research and Quality patient safety indicators (PSIs) in the United States before and during the first surge of the pandemic among patients admitted without COVID-19.

METHODS

We analyzed trends in PSIs from January 2019 to June 2020 in patients without COVID-19 using data from IBM MarketScan Commercial Database. We included members of employer-sponsored or Medicare supplemental health plans with inpatient, non-COVID-19 admissions. The primary outcomes were risk-adjusted composite and individual PSIs.

RESULTS

We analyzed 1,869,430 patients admitted without COVID-19. Among patients without COVID-19, the composite PSI score was not significantly different when comparing the first surge (Q2 2020) to the prepandemic period (e.g., Q2 2020 score of 2.46 [95% confidence interval {CI}, 2.34-2.58] versus Q1 2020 score of 2.37 [95% CI, 2.27-2.46]; P = 0.22). Individual PSIs for these patients during Q2 2020 were also not significantly different, except in-hospital fall with hip fracture (e.g., Q2 2020 was 3.42 [95% CI, 3.34-3.49] versus Q4 2019 was 2.45 [95% CI, 2.40-2.50]; P = 0.01).

CONCLUSIONS

The first surge of COVID-19 was not associated with worse inpatient safety for patients without COVID-19, highlighting the ability of the healthcare system to respond to the initial surge of the pandemic.

Clinical Decision Support for Hypertension Management in Chronic Kidney Disease: A Randomized Clinical Trial

Importance

Chronic kidney disease (CKD) affects 37 million adults in the United States, and for patients with CKD, hypertension is a key risk factor for adverse outcomes, such as kidney failure, cardiovascular events, and death.

Objective

To evaluate a computerized clinical decision support (CDS) system for the management of uncontrolled hypertension in patients with CKD.

Design, Setting, and Participants

This multiclinic, randomized clinical trial randomized primary care practitioners (PCPs) at a primary care network, including 15 hospital-based, ambulatory, and community health center-based clinics, through a stratified, matched-pair randomization approach February 2021 to February 2022. All adult patients with a visit to a PCP in the last 2 years were eligible and those with evidence of CKD and hypertension were included.

Intervention

The intervention consisted of a CDS system based on behavioral economic principles and human-centered design methods that delivered tailored, evidence-based recommendations, including initiation or titration of renin-angiotensin-aldosterone system inhibitors. The patients in the control group received usual care from PCPs with the CDS system operating in silent mode.

Main Outcomes and Measures

The primary outcome was the change in mean systolic blood pressure (SBP) between baseline and 180 days compared between groups. The primary analysis was a repeated measures linear mixed model, using SBP at baseline, 90 days, and 180 days in an intention-to-treat repeated measures model to account for missing data. Secondary outcomes included blood pressure (BP) control and outcomes such as percentage of patients who received an action that aligned with the CDS recommendations.

Results

The study included 174 PCPs and 2026 patients (mean [SD] age, 75.3 [0.3] years; 1223 [60.4%] female; mean [SD] SBP at baseline, 154.0 [14.3] mm Hg), with 87 PCPs and 1029 patients randomized to the intervention and 87 PCPs and 997 patients randomized to usual care. Overall, 1714 patients (84.6%) were treated for hypertension at baseline. There were 1623 patients (80.1%) with an SBP measurement at 180 days. From the linear mixed model, there was a statistically significant difference in mean SBP change in the intervention group compared with the usual care group (change, -14.6 [95% CI, -13.1 to -16.0] mm Hg vs -11.7 [-10.2 to -13.1] mm Hg; P = .005). There was no difference in the percentage of patients who achieved BP control in the intervention group compared with the control group (50.4% [95% CI, 46.5% to 54.3%] vs 47.1% [95% CI, 43.3% to 51.0%]). More patients received an action aligned with the CDS recommendations in the intervention group than in the usual care group (49.9% [95% CI, 45.1% to 54.8%] vs 34.6% [95% CI, 29.8% to 39.4%]; P < .001).

Conclusions and Relevance

These findings suggest that implementing this computerized CDS system could lead to improved management of uncontrolled hypertension and potentially improved clinical outcomes at the population level for patients with CKD.

Trial Registration

ClinicalTrials.gov Identifier: NCT03679247.

Stimulant Drugs and Stimulant Use Disorder

The authors aim to summarize several key points of stimulant drugs and stimulant use disorder, including their indications, short-term and long-term adverse effects, current treatment strategies, and association with opioid medications. The global prevalence of stimulant use has seen annual increase in the last decade. Multiple studies have shown that stimulant use and stimulant use disorder are associated with a range of individual and public health issues. Stimulant misuse has led to a significant increase of overdose deaths in the United States.

Patient Safety and Artificial Intelligence in Clinical Care

This Viewpoint offers 3 recommendations for health care organizations and other stakeholders to consider as part of the Health and Human Services’ artificial intelligence safety program.

Deep Survival Analysis for Interpretable Time-Varying Prediction of Preeclampsia Risk

Objective

Survival analysis is widely utilized in healthcare to predict the timing of disease onset. Traditional methods of survival analysis are usually based on Cox Proportional Hazards model and assume proportional risk for all subjects. However, this assumption is rarely true for most diseases, as the underlying factors have complex, non-linear, and time-varying relationships. This concern is especially relevant for pregnancy, where the risk for pregnancy-related complications, such as preeclampsia, varies across gestation. Recently, deep learning survival models have shown promise in addressing the limitations of classical models, as the novel models allow for non-proportional risk handling, capturing nonlinear relationships, and navigating complex temporal dynamics.

Methods

We present a methodology to model the temporal risk of preeclampsia during pregnancy and investigate the associated clinical risk factors. We utilized a retrospective dataset including 66,425 pregnant individuals who delivered in two tertiary care centers from 2015-2023. We modeled the preeclampsia risk by modifying DeepHit, a deep survival model, which leverages neural network architecture to capture time-varying relationships between covariates in pregnancy. We applied time series k-means clustering to DeepHit's normalized output and investigated interpretability using Shapley values.

Results

We demonstrate that DeepHit can effectively handle high-dimensional data and evolving risk hazards over time with performance similar to the Cox Proportional Hazards model, achieving an area under the curve (AUC) of 0.78 for both models. The deep survival model outperformed traditional methodology by identifying time-varied risk trajectories for preeclampsia, providing insights for early and individualized intervention. K-means clustering resulted in patients delineating into low-risk, early-onset, and late-onset preeclampsia groups- notably, each of those has distinct risk factors.

Conclusion

This work demonstrates a novel application of deep survival analysis in time-varying prediction of preeclampsia risk. Our results highlight the advantage of deep survival models compared to Cox Proportional Hazards models in providing personalized risk trajectory and demonstrating the potential of deep survival models to generate interpretable and meaningful clinical applications in medicine.

Testing of an Electronic Clinical Quality Measure for Diagnostic Delay of Venous Thromboembolism (DOVE) in Primary Care

Venous Thromboembolism (VTE) is a serious, preventable public health problem that requires timely treatment. Because signs and symptoms are non-specific, patients often present to primary care providers with VTE symptoms prior to diagnosis. Today there are no federal measurement tools in place to track delayed diagnosis of VTE. We developed and tested an electronic clinical quality measure (eCQM) to quantify Diagnostic Delay of Venous Thromboembolism (DOVE); the rate of avoidable delayed VTE events occurring in patients with a VTE who had reported VTE symptoms in primary care within 30 days of diagnosis. DOVE uses routinely collected EHR data without contributing to documentation burden. DOVE was tested in two geographically distant healthcare systems. Overall DOVE rates were 72.60% (site 1) and 77.14% (site 2). This novel, data-driven eCQM could inform healthcare providers and facilities about opportunities to improve care, strengthen incentives for quality improvement, and ultimately improve patient safety.

Adverse Events in Pediatric Inpatients: The Japan Adverse Event Study

OBJECTIVES

Adverse events (AEs) represent an important cause of morbidity and mortality for pediatric inpatients; however, reports on their epidemiology in pediatrics, especially outside Western countries, are scarce. We investigated the incidence and nature of AEs in pediatric inpatients in Japan.

METHODS

Trained pediatrician and pediatric nurses reviewed all medical documents of 1126 pediatric inpatients in 2 tertiary care teaching hospitals in Japan, and potential incidents were collected with patients' characteristics. Age was categorized into 6 groups (neonates, infants, preschoolers, school-aged children, teenagers, and over-aged pediatric patients), and medical care when potential incidents occurred was classified into drug, operation, procedure/examinations, nursing, management, and judgment. Physician reviewers independently evaluated all collected incidents into AEs, potential AEs, medical errors, and exclusions and assessed their severity and preventability.

RESULTS

A total of 1126 patients with 12,624 patient-days were enrolled, and 953 AEs, with an incidence of 76 (95% confidence interval, 71-80) per 1000 patient-days, were identified. Preventable AEs accounted for 23% (218/953) of AEs. The incidence of AEs tended to decrease with increasing age. The proportion of AEs that were preventable was highest in neonates (40%), and this proportion decreased as children aged. Both judgment and management-related AEs were considered preventable AEs, and judgment-related AEs were more severe AEs than no-judgment-related AEs; 43% were life-threatening.

CONCLUSIONS

Adverse events were common in Japanese pediatric inpatients, and their preventability and severity varied considerably by age category and medical care. Further investigation is needed to address which strategies might most improve pediatric patient safety.

Intravenous Medication Administration Safety with Smart Infusion Pumps in the Neonatal Intensive Care Unit: An Observational Study

INTRODUCTION

Infants in the neonatal intensive care unit (NICU) are among the most vulnerable patient populations and medication errors are a significant source of risk and harm to neonates. Smart infusion pumps have been implemented to support the safe medication administration process; however, the effect of using smart infusion pumps on medication safety in the NICU is still unclear.

METHODS

We conducted an observational study with a prospective point-prevalence approach to investigate intravenous (IV) medication administration errors in the NICU at one academic medical center in the USA. Observations were conducted in 48 days in a 3-month data collection period in 2019.

RESULTS

We observed a total of 441 patients with 905 IV medication administrations during the data collection period. The total number of errors was 130 (14.4 per 100 administrations). Of these, the most frequent errors were selecting the wrong drug library entry (5.3 per 100 administrations), unauthorized medication (0.7 per 100 administrations), and wrong dose (0.6 per 100 administrations). Sixty-eight errors (7.5 per 100 administrations) were unlikely to cause harm despite reaching the patient (category C errors), while the rest did not reach the patient.

CONCLUSION

We identified the medication errors, which was unique to NICU populations, but no harm to the patients were identified. Most errors occurred due to a lack of compliance of using smart pump technology; therefore, potential exists to maximize safety related to medication administration practices in the NICU through hospital policy change and increasing adherence to appropriate use of smart pump technology.

Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study

BACKGROUND

Large language models (LLMs) such as GPT-4 hold great promise as transformative tools in health care, ranging from automating administrative tasks to augmenting clinical decision making. However, these models also pose a danger of perpetuating biases and delivering incorrect medical diagnoses, which can have a direct, harmful impact on medical care. We aimed to assess whether GPT-4 encodes racial and gender biases that impact its use in health care.

METHODS

Using the Azure OpenAI application interface, this model evaluation study tested whether GPT-4 encodes racial and gender biases and examined the impact of such biases on four potential applications of LLMs in the clinical domain-namely, medical education, diagnostic reasoning, clinical plan generation, and subjective patient assessment. We conducted experiments with prompts designed to resemble typical use of GPT-4 within clinical and medical education applications. We used clinical vignettes from NEJM Healer and from published research on implicit bias in health care. GPT-4 estimates of the demographic distribution of medical conditions were compared with true US prevalence estimates. Differential diagnosis and treatment planning were evaluated across demographic groups using standard statistical tests for significance between groups.

FINDINGS

We found that GPT-4 did not appropriately model the demographic diversity of medical conditions, consistently producing clinical vignettes that stereotype demographic presentations. The differential diagnoses created by GPT-4 for standardised clinical vignettes were more likely to include diagnoses that stereotype certain races, ethnicities, and genders. Assessment and plans created by the model showed significant association between demographic attributes and recommendations for more expensive procedures as well as differences in patient perception.

INTERPRETATION

Our findings highlight the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care. We discuss the potential sources of these biases and potential mitigation strategies before clinical implementation.

FUNDING

Priscilla Chan and Mark Zuckerberg.

Advancing health information during interhospital transfer: An interrupted time series

INTRODUCTION

Although the transfer of patients between acute care hospitals (interhospital transfer, IHT) is common, health information exchange (HIE) during IHT remains inadequate, with fragmented communication and unreliable access to clinical information. This study aims to design, implement, and rigorously evaluate the implementation of a HIE platform to improve data access during IHT.

METHODS AND ANALYSIS

Study subjects include patients aged >18 transferred to the medical, cardiology, oncology, or intensive care unit (ICU) services at an 800-bed quaternary care hospital; and healthcare workers involved in their care. The first aim of this study is to optimize clinician workflow, data visualization, and interoperability through user-centered design sessions for HIE platform development. The second aim is to evaluate the impact of the intervention on clinician-reported medical errors among 500 pre- and 500 postintervention IHT patients using interrupted time series methodology, adjusting for confounding variables and temporal trends. The third aim is to evaluate intervention fidelity, use and perceived usability of the platform, and barriers and facilitators of implementation from interprofessional stakeholder input, using mixed-methods evaluation. The fourth aim is to consolidate key findings to create a toolkit for spread and sustainability.

ETHICS AND DISSEMINATION

We will track patient safety endpoints and clinician workflow burdens and ensure the protection of patient data throughout the study. We will disseminate our findings via the creation of a toolkit for spread and sustainability, partnering with our funder (AHRQ) for dissemination, and communicating our results via abstracts and publications.

Cost Savings Associated With Implementing 4 Total Joint Replacement Electronic Clinical Quality Measures Nationally: 2020-2040

BACKGROUND

Digital transformation using widely available electronic data is a key component to improving health outcomes and customer choice and decreasing cost and measurement burden. Despite these benefits, existing information on the potential cost savings from electronic clinical quality measures (eCQMs) is limited.

METHODS

We assessed the costs of implementing 4 eCQMs related to total hip and/or total knee arthroplasty into electronic health record systems across healthcare systems in the United States. We used published literature and technical expert panel consultation to calculate low-, mid-, and high-range hip and knee arthroplasty surgery projections, and used empirical testing, literature, and technical expert panel consultation to develop an economic model to assess projected cost savings of eCQMs when implemented nationally.

RESULTS

Low-, mid-, and high-range projected cost savings for year's 2020, 2030, and 2040 were calculated for 4 orthopedic eCQMs. Mid-range projected cost savings for 2020 ranged from $7.9 to $31.9 million per measure per year. A breakeven of between 0.5% and 5.1% of adverse events (measure dependent) must be averted for cost savings to outweigh implementation costs.

CONCLUSIONS

All measures demonstrated potential cost savings. These findings suggest that eCQMs have the potential to lower healthcare costs and improve patient outcomes without adding to physician documentation burden. The Centers for Medicare and Medicaid Services' investment in eCQMs is an opportunity to reduce adverse outcomes and excess costs in orthopedics.

Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models

Many areas of medicine would benefit from deeper, more accurate phenotyping, but there are limited approaches for phenotyping using clinical notes without substantial annotated data. Large language models (LLMs) have demonstrated immense potential to adapt to novel tasks with no additional training by specifying task-specific instructions. Here we report the performance of a publicly available LLM, Flan-T5, in phenotyping patients with postpartum hemorrhage (PPH) using discharge notes from electronic health records (n = 271,081). The language model achieves strong performance in extracting 24 granular concepts associated with PPH. Identifying these granular concepts accurately allows the development of interpretable, complex phenotypes and subtypes. The Flan-T5 model achieves high fidelity in phenotyping PPH (positive predictive value of 0.95), identifying 47% more patients with this complication compared to the current standard of using claims codes. This LLM pipeline can be used reliably for subtyping PPH and outperforms a claims-based approach on the three most common PPH subtypes associated with uterine atony, abnormal placentation, and obstetric trauma. The advantage of this approach to subtyping is its interpretability, as each concept contributing to the subtype determination can be evaluated. Moreover, as definitions may change over time due to new guidelines, using granular concepts to create complex phenotypes enables prompt and efficient updating of the algorithm. Using this language modelling approach enables rapid phenotyping without the need for any manually annotated training data across multiple clinical use cases.

The Electronic Inbox-Benefits, Questions, and Solutions for the Road Ahead

This Viewpoint looks at digital communication between patients and physicians, including approaches to provide adequate support for these efforts that balance patient needs with appropriate time investments from clinicians.

Implementation of a Continuous Patient Monitoring System in the Hospital Setting: A Qualitative Study

BACKGROUND

Technology can improve care delivery, patient outcomes, and staff satisfaction, but integration into the clinical workflow remains challenging. To contribute to this knowledge area, this study examined the implementation continuum of a contact-free, continuous monitoring system (CFCM) in an inpatient setting. CFCM monitors vital signs and uses the information to alert clinicians of important changes, enabling early detection of patient deterioration.

METHODS

Data were collected throughout the entire implementation continuum at a community teaching hospital. Throughout the study, 3 group and 24 individual interviews and five process observations were conducted. Postimplementation alarm response data were collected. Analysis was conducted using triangulation of information sources and two-coder consensus.

RESULTS

Preimplementation perceived barriers were alarm fatigue, questions about accuracy and trust, impact on patient experience, and challenges to the status quo. Stakeholders identified the value of CFCM as preventing deterioration and benefitting patients who are not good candidates for telemetry. Educational materials addressed each barrier and emphasized the shared CFCM values. Mean alarm response times were below the desired target of two minutes. Postimplementation interview analysis themes revealed lessened concerns of alarm fatigue and improved trust in CFCM than anticipated. Postimplementation challenges included insufficient training for secondary users and impact on patient experience.

CONCLUSION

In addition to understanding the preimplementation anticipated barriers to implementation and establishing shared value before implementation, future recommendations include studying strategies for optimal tailoring of education to each user group, identifying and reinforcing positive process changes after implementation, and including patient experience as the overarching element in frameworks for digital tool implementation.

A deep learning approach for transgender and gender diverse patient identification in electronic health records

BACKGROUND

Although accurate identification of gender identity in the electronic health record (EHR) is crucial for providing equitable health care, particularly for transgender and gender diverse (TGD) populations, it remains a challenging task due to incomplete gender information in structured EHR fields.

OBJECTIVE

Using TGD identification as a case study, this research uses NLP and deep learning to build an accurate patient gender identity predictive model, aiming to tackle the challenges of identifying relevant patient-level information from EHR data and reducing annotation work.

METHODS

This study included adult patients in a large healthcare system in Boston, MA, between 4/1/2017 to 4/1/2022. To identify relevant information from massive clinical notes, we compiled a list of gender-related keywords through expert curation, literature review, and expansion via a fine-tuned BioWordVec model. This keyword list was used to pre-screen potential TGD individuals and create two datasets for model training, testing, and validation. Dataset I was a balanced dataset that contained clinician-confirmed TGD patients and cases without keywords. Dataset II contained cases with keywords. The performance of the deep learning model was compared to traditional machine learning and rule-based algorithms.

RESULTS

The final keyword list consists of 109 keywords, of which 58 (53.2%) were expanded by the BioWordVec model. Dataset I contained 3,150 patients (50% TGD) while Dataset II contained 200 patients (90% TGD). On Dataset I the deep learning model achieved a F1 score of 0.917, sensitivity of 0.854, and a precision of 0.980; and on Dataset II a F1 score of 0.969, sensitivity of 0.967, and precision of 0.972. The deep learning model significantly outperformed rule-based algorithms.

CONCLUSION

This is the first study to show that deep learning-integrated NLP algorithms can accurately identify gender identity using EHR data. Future work should leverage and evaluate additional diverse data sources to generate more generalizable algorithms.

System-Level Factors and Time Spent on Electronic Health Records by Primary Care Physicians

Importance

Primary care physicians (PCPs) spend the most time on the electronic health record (EHR) of any specialty. Thus, it is critical to understand what factors contribute to varying levels of PCP time spent on EHRs.

Objective

To characterize variation in EHR time across PCPs and primary care clinics, and to describe how specific PCP, patient panel, clinic, and team collaboration factors are associated with PCPs' time spent on EHRs.

Design, Setting, and Participants

This cross-sectional study included 307 PCPs practicing across 31 primary care clinics at Massachusetts General Hospital and Brigham and Women's Hospital during 2021. Data were analyzed from October 2022 to October 2023.

Main Outcomes and Measures

Total per-visit EHR time, total per-visit pajama time (ie, time spent on the EHR between 5:30 pm to 7:00 am and on weekends), and total per-visit time on the electronic inbox as measured by activity log data derived from an EHR database.

Results

The sample included 307 PCPs (183 [59.6%] female). On a per-visit basis, PCPs spent a median (IQR) of 36.2 (28.9-45.7) total minutes on the EHR, 6.2 (3.1-11.5) minutes of pajama time, and 7.8 (5.5-10.7) minutes on the electronic inbox. When comparing PCP time expenditure by clinic, median (IQR) total EHR time, median (IQR) pajama time, and median (IQR) electronic inbox time ranged from 23.5 (20.7-53.1) to 47.9 (30.6-70.7) minutes per visit, 1.7 (0.7-10.5) to 13.1 (7.7-28.2) minutes per visit, and 4.7 (4.1-5.2) to 10.8 (8.9-15.2) minutes per visit, respectively. In a multivariable model with an outcome of total per-visit EHR time per visit, an above median percentage of teamwork on orders was associated with 3.81 (95% CI, 0.49-7.13) minutes per visit fewer and having a clinic pharmacy technician was associated with 7.87 (95% CI, 2.03-13.72) minutes per visit fewer. Practicing in a community health center was associated with fewer minutes of total EHR time per visit (5.40 [95% CI, 0.06-10.74] minutes).

Conclusions and Relevance

There is substantial variation in EHR time among individual PCPs and PCPs within clinics. Organization-level factors, such as team collaboration on orders, support for medication refill functions, and practicing in a community health center, are associated with lower EHR time for PCPs. These findings highlight the importance of addressing EHR burden at a systems level.

A resilience perspective on healthcare personnels' experiences of managing the COVID-19 pandemic: a qualitative study in Norwegian nursing homes and come care services

BACKGROUND

The COVID-19 pandemic led to new and unfamiliar changes in healthcare services globally. Most COVID-19 patients were cared for in primary healthcare services, demanding major adjustments and adaptations in care delivery. Research addressing how rural primary healthcare services coped during the COVID-19 pandemic, and the possible learning potential originating from the pandemic is limited. The aim of this study was to assess how primary healthcare personnel (PHCP) working in rural areas experienced the work situation during the COVID-19 outbreak, and how adaptations to changes induced by the pandemic were handled in nursing homes and home care services.

METHOD

This study was conducted as an explorative qualitative study. Four municipalities with affiliated nursing homes and homecare services were included in the study. We conducted focus group interviews with primary healthcare personnel working in rural nursing homes and homecare services in western Norway. The included PHCP were 16 nurses, 7 assistant nurses and 2 assistants. Interviews were audio recorded, transcribed and analyzed using thematic analysis.

RESULTS

The analysis resulted in three main themes and 16 subthemes describing PHCP experience of the work situation during the COVID-19 pandemic, and how they adapted to the changes and challenges induced by the pandemic. The main themes were: "PHCP demonstrated high adaptive capacity while being put to the test", "Adapting to organizational measures, with varying degree of success" and "Safeguarding the patient's safety and quality of care, but at certain costs".

CONCLUSION

This study demonstrated PHCPs major adaptive capacity in response to the challenges and changes induced by the covid-19 pandemic, while working under varying organizational conditions. Many adaptations where long-term solutions improving healthcare delivery, others where short-term solutions forced by inadequate management, governance, or a lack of leadership. Overall, the findings demonstrated the need for all parts of the system to engage in building resilient healthcare services. More research investigating this learning potential, particularly in primary healthcare services, is needed.

How Safe are Outpatient Electronic Health Records? An Evaluation of Medication-Related Decision Support using the Ambulatory Electronic Health Record Evaluation Tool

BACKGROUND

The purpose of the Ambulatory Electronic Health Record (EHR) Evaluation Tool is to provide outpatient clinics with an assessment that they can use to measure the ability of the EHR system to detect and prevent common prescriber errors. The tool consists of a medication safety test and a medication reconciliation module.

OBJECTIVES

The goal of this study was to perform a broad evaluation of outpatient medication-related decision support using the Ambulatory EHR Evaluation Tool.

METHODS

We performed a cross-sectional study with 10 outpatient clinics using the Ambulatory EHR Evaluation Tool. For the medication safety test, clinics were provided test patients and associated medication test orders to enter in their EHR, where they recorded any advice or information they received. Once finished, clinics received an overall percentage score of unsafe orders detected and individual order category scores. For the medication reconciliation module, clinics were asked to electronically reconcile two medication lists, where modifications were made by adding and removing medications and changing the dosage of select medications.

RESULTS

For the medication safety test, the mean overall score was 57%, with the highest score being 70%, and the lowest score being 40%. Clinics performed well in the drug allergy (100%), drug dose daily (85%), and inappropriate medication combinations (74%) order categories. Order categories with the lowest performance were drug laboratory (10%) and drug monitoring (3%). Most clinics (90%) scored a 0% in at least one order category. For the medication reconciliation module, only one clinic (10%) could reconcile medication lists electronically; however, there was no clinical decision support available that checked for drug interactions.

CONCLUSION

We evaluated a sample of ambulatory practices around their medication-related decision support and found that advanced capabilities within these systems have yet to be widely implemented. The tool was practical to use and identified substantial opportunities for improvement in outpatient medication safety.

Factors determining safety culture in hospitals: a scoping review

OBJECTIVE

To evaluate and synthesise the factors determining patient safety culture in hospitals.

METHODS

The scoping review protocol was based on the criteria of the Joanna Briggs Institute. Eligibility criteria were as follows: (1) empirical study published in a peer-reviewed journal; (2) used methods or tools to assess, study or measure safety culture or climate; (3) data collected in the hospital setting and (4) studies published in English. Relevant literature was located using PubMed, CINAHL, Web of Science and PsycINFO databases. Quantitative and qualitative analyses were performed using RStudio and the R interface for multidimensional analysis of texts and questionnaires (IRaMuTeQ).

RESULTS

A total of 248 primary studies were included. The most used instruments for assessing safety culture were the Hospital Survey on Patient Safety Culture (n=104) and the Safety Attitudes Questionnaire (n=63). The Maslach Burnout Inventory (n=13) and Culture Assessment Scales based on patient perception (n=9) were used in association with cultural instruments. Sixty-six articles were included in the qualitative analysis. In word cloud and similarity analyses, the words 'communication' and 'leadership' were most prominent. Regarding the descending hierarchical classification analysis, the content was categorised into two main classes, one of which was subdivided into five subclasses: class 1a: job satisfaction and leadership (15.56%), class 1b: error response (22.22%), class 1c: psychological and empowerment nurses (20.00%), class 1d: trust culture (22.22%) and class 2: innovation worker (20.00%).

CONCLUSION

The instruments presented elements that remained indispensable for assessing the safety culture, such as leadership commitment, open communication and learning from mistakes. There was also a tendency for research to assess patient and family engagement, psychological safety, nurses' engagement in decision-making and innovation.

Inpatient EHR User Experience and Hospital EHR Safety Performance

IMPORTANCE

Despite the broad adoption and optimization of electronic health record (EHR) systems across the continuum of care, serious usability and safety problems persist.

OBJECTIVE

To assess whether EHR safety performance is associated with EHR frontline user experience in a national sample of hospitals.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study included all US adult hospitals that used the National Quality Forum Leapfrog Health IT Safety Measure and also used the ARCH Collaborative EHR User experience survey from January 1, 2017, to January 1, 2019. Data analysis was performed from September 2020 to November 2022.

MAIN OUTCOMES AND MEASURES

The primary outcomes were hospital performance on the Leapfrog Health IT Safety measure (overall and 10 subcomponents) and the ARCH collaborative frontline user experience scores (overall and 8 subcomponents). Ordinary least squares models with survey responses clustered by hospital were used to assess associations between the overall measures and their subcomponents.

RESULTS

There were 112 hospitals and 5689 frontline user surveys included in the study. Hospitals scored a mean of 0.673 (range, 0.297-0.973) on the Leapfrog Health IT safety measure; the mean ARCH EHR user experience score was 3.377 (range, 1 [best] to 5 [worst]). The adjusted β coefficient between the overall safety score and overall user experience score was 0.011 (95% CI, 0.006-0.016). The ARCH overall score was also significantly associated with 10 subcategory scores of the Leapfrog Health IT safety score, and the overall Leapfrog score was associated with the 8 subcategory scores of the ARCH user experience score.

CONCLUSIONS AND RELEVANCE

This cross-sectional study found a positive association between frontline user-rated EHR usability and EHR safety performance. This finding suggests that improving EHR usability, which is a current well-known pain point for EHR users, could have direct benefits in terms of improved EHR safety.

Facilitators and Barriers to Interacting With Clinical Decision Support in the ICU: A Mixed-Methods Approach

OBJECTIVES

Clinical decision support systems (CDSSs) are used in various aspects of healthcare to improve clinical decision-making, including in the ICU. However, there is growing evidence that CDSS are not used to their full potential, often resulting in alert fatigue which has been associated with patient harm. Clinicians in the ICU may be more vulnerable to desensitization of alerts than clinicians in less urgent parts of the hospital. We evaluated facilitators and barriers to appropriate CDSS interaction and provide methods to improve currently available CDSS in the ICU.

DESIGN

Sequential explanatory mixed-methods study design, using the BEhavior and Acceptance fRamework.

SETTING

International survey study.

PATIENT/SUBJECTS

Clinicians (pharmacists, physicians) identified via survey, with recent experience with clinical decision support.

INTERVENTIONS

An initial survey was developed to evaluate clinician perspectives on their interactions with CDSS. A subsequent in-depth interview was developed to further evaluate clinician (pharmacist, physician) beliefs and behaviors about CDSS. These interviews were then qualitatively analyzed to determine themes of facilitators and barriers with CDSS interactions.

MEASUREMENTS AND MAIN RESULTS

A total of 48 respondents completed the initial survey (estimated response rate 15.5%). The majority believed that responding to CDSS alerts was part of their job (75%) but felt they experienced alert fatigue (56.5%). In the qualitative analysis, a total of five facilitators (patient safety, ease of response, specificity, prioritization, and feedback) and four barriers (excess quantity, work environment, difficulty in response, and irrelevance) were identified from the in-depth interviews.

CONCLUSIONS

In this mixed-methods survey, we identified areas that institutions should focus on to improve appropriate clinician interactions with CDSS, specific to the ICU. Tailoring of CDSS to the ICU may lead to improvement in CDSS and subsequent improved patient safety outcomes.

An Interpretable Longitudinal Preeclampsia Risk Prediction Using Machine Learning

Background

Preeclampsia is a pregnancy-specific disease characterized by new onset hypertension after 20 weeks of gestation that affects 2-8% of all pregnancies and contributes to up to 26% of maternal deaths. Despite extensive clinical research, current predictive tools fail to identify up to 66% of patients who will develop preeclampsia. We sought to develop a tool to longitudinally predict preeclampsia risk.

Methods

In this retrospective model development and validation study, we examined a large cohort of patients who delivered at six community and two tertiary care hospitals in the New England region between 02/2015 and 06/2023. We used sociodemographic, clinical diagnoses, family history, laboratory, and vital signs data. We developed eight datasets at 14, 20, 24, 28, 32, 36, 39 weeks gestation and at the hospital admission for delivery. We created linear regression, random forest, xgboost, and deep neural networks to develop multiple models and compared their performance. We used Shapley values to investigate the global and local explainability of the models and the relationships between the predictive variables.

Findings

Our study population (N=120,752) had an incidence of preeclampsia of 5.7% (N=6,920). The performance of the models as measured using the area under the curve, AUC, was in the range 0.73-0.91, which was externally validated. The relationships between some of the variables were complex and non-linear; in addition, the relative significance of the predictors varied over the pregnancy. Compared to the current standard of care for preeclampsia risk stratification in the first trimester, our model would allow 48.6% more at-risk patients to be identified.

Interpretation

Our novel preeclampsia prediction tool would allow clinicians to identify patients at risk early and provide personalized predictions, as well as longitudinal predictions throughout pregnancy.

Funding

National Institutes of Health, Anesthesia Patient Safety Foundation.

RESEARCH IN CONTEXT

Evidence before this study: Current tools for the prediction of preeclampsia are lacking as they fail to identify up to 66% of the patients who develop preeclampsia. We searched PubMed, MEDLINE, and the Web of Science from database inception to May 1, 2023, using the keywords "deep learning", "machine learning", "preeclampsia", "artificial intelligence", "pregnancy complications", and "predictive models". We identified 13 studies that employed machine learning to develop prediction models for preeclampsia risk based on clinical variables. Among these studies, six included biomarkers such as serum placental growth factor, pregnancy-associated plasma protein A, and uterine artery pulsatility index, which are not routinely available in our clinical practice; two studies were in diverse cohorts of more than 100 000 patients, and two studies developed longitudinal predictions using medical records data. However, most studies have limited depth, concerns about data leakage, overfitting, or lack of generalizability.Added value of this study: We developed a comprehensive longitudinal predictive tool based on routine clinical data that can be used throughout pregnancy to predict the risk of preeclampsia. We tested multiple types of predictive models, including machine learning and deep learning models, and demonstrated high predictive power. We investigated the changes over different time points of individual and group variables and found previously known and novel relationships between variables such as red blood cell count and preeclampsia risk.Implications of all the available evidence: Longitudinal prediction of preeclampsia using machine learning can be achieved with high performance. Implementation of an accurate predictive tool within the electronic health records can aid clinical care and identify patients at heightened risk who would benefit from aspirin prophylaxis, increased surveillance, early diagnosis, and escalation in care. These results highlight the potential of using artificial intelligence in clinical decision support, with the ultimate goal of reducing iatrogenic preterm birth and improving perinatal care.

Road map for clinicians to develop and evaluate AI predictive models to inform clinical decision-making

BACKGROUND

Predictive models have been used in clinical care for decades. They can determine the risk of a patient developing a particular condition or complication and inform the shared decision-making process. Developing artificial intelligence (AI) predictive models for use in clinical practice is challenging; even if they have good predictive performance, this does not guarantee that they will be used or enhance decision-making. We describe nine stages of developing and evaluating a predictive AI model, recognising the challenges that clinicians might face at each stage and providing practical tips to help manage them.

FINDINGS

The nine stages included clarifying the clinical question or outcome(s) of interest (output), identifying appropriate predictors (features selection), choosing relevant datasets, developing the AI predictive model, validating and testing the developed model, presenting and interpreting the model prediction(s), licensing and maintaining the AI predictive model and evaluating the impact of the AI predictive model. The introduction of an AI prediction model into clinical practice usually consists of multiple interacting components, including the accuracy of the model predictions, physician and patient understanding and use of these probabilities, expected effectiveness of subsequent actions or interventions and adherence to these. Much of the difference in whether benefits are realised relates to whether the predictions are given to clinicians in a timely way that enables them to take an appropriate action.

CONCLUSION

The downstream effects on processes and outcomes of AI prediction models vary widely, and it is essential to evaluate the use in clinical practice using an appropriate study design.

A Multiyear Survey Evaluating Clinician Electronic Health Record Satisfaction

OBJECTIVES

We assessed how clinician satisfaction with a vendor electronic health record (EHR) changed over time in the 4 years following the transition from a homegrown EHR system to identify areas for improvement.

METHODS

We conducted a multiyear survey of clinicians across a large health care system after transitioning to a vendor EHR. Eligible clinicians from the first institution to transition received a survey invitation by email in fall 2016 and then eligible clinicians systemwide received surveys in spring 2018 and spring 2019. The survey included items assessing ease/difficulty of completing tasks and items assessing perceptions of the EHR's value, usability, and impact. One item assessing overall satisfaction and one open-ended question were included. Frequencies and means were calculated, and comparison of means was performed between 2018 and 2019 on all clinicians. A multivariable generalized linear model was performed to predict the outcome of overall satisfaction.

RESULTS

Response rates for the surveys ranged from 14 to 19%. The mean response from 3 years of surveys for one institution, Brigham and Women's Hospital, increased for overall satisfaction between 2016 (2.85), 2018 (3.01), and 2019 (3.21, p < 0.001). We found no significant differences in mean response for overall satisfaction between all responders of the 2018 survey (3.14) and those of the 2019 survey (3.19). Systemwide, tasks rated the most difficult included "Monitoring patient medication adherence," "Identifying when a referral has not been completed," and "Making a list of patients based on clinical information (e.g., problem, medication)." Clinicians disagreed the most with "The EHR helps me focus on patient care rather than the computer" and "The EHR allows me to complete tasks efficiently."

CONCLUSION

Survey results indicate room for improvement in clinician satisfaction with the EHR. Usability of EHRs should continue to be an area of focus to ease clinician burden and improve clinician experience.

Towards Medical Billing Automation: NLP for Outpatient Clinician Note Classification

Objectives

Our primary objective was to develop a natural language processing approach that accurately predicts outpatient Evaluation and Management (E/M) level of service (LoS) codes using clinicians' notes from a health system electronic health record. A secondary objective was to investigate the impact of clinic note de-identification on document classification performance.

Methods

We used retrospective outpatient office clinic notes from four medical and surgical specialties. Classification models were fine-tuned on the clinic notes datasets and stratified by subspecialty. The success criteria for the classification tasks were the classification accuracy and F1-scores on internal test data. For the secondary objective, the dataset was de-identified using Named Entity Recognition (NER) to remove protected health information (PHI), and models were retrained.

Results

The models demonstrated similar predictive performance across different specialties, except for internal medicine, which had the lowest classification accuracy across all model architectures. The models trained on the entire note corpus achieved an E/M LoS CPT code classification accuracy of 74.8% (CI 95: 74.1-75.6). However, the de-identified note corpus showed a markedly lower classification accuracy of 48.2% (CI 95: 47.7-48.6) compared to the model trained on the identified notes.

Conclusion

The study demonstrates the potential of NLP-based document classifiers to accurately predict E/M LoS CPT codes using clinical notes from various medical and procedural specialties. The models' performance suggests that the classification task's complexity merits further investigation. The de-identification experiment demonstrated that de-identification may negatively impact classifier performance. Further research is needed to validate the performance of our NLP classifiers in different healthcare settings and patient populations and to investigate the potential implications of de-identification on model performance.

Developing, pilot testing, and refining requirements for 3 EHR-integrated interventions to improve diagnostic safety in acute care: a user-centered approach

Objective

To describe a user-centered approach to develop, pilot test, and refine requirements for 3 electronic health record (EHR)-integrated interventions that target key diagnostic process failures in hospitalized patients.

Materials and Methods

Three interventions were prioritized for development: a Diagnostic Safety Column (DSC) within an EHR-integrated dashboard to identify at-risk patients; a Diagnostic Time-Out (DTO) for clinicians to reassess the working diagnosis; and a Patient Diagnosis Questionnaire (PDQ) to gather patient concerns about the diagnostic process. Initial requirements were refined from analysis of test cases with elevated risk predicted by DSC logic compared to risk perceived by a clinician working group; DTO testing sessions with clinicians; PDQ responses from patients; and focus groups with clinicians and patient advisors using storyboarding to model the integrated interventions. Mixed methods analysis of participant responses was used to identify final requirements and potential implementation barriers.

Results

Final requirements from analysis of 10 test cases predicted by the DSC, 18 clinician DTO participants, and 39 PDQ responses included the following: DSC configurable parameters (variables, weights) to adjust baseline risk estimates in real-time based on new clinical data collected during hospitalization; more concise DTO wording and flexibility for clinicians to conduct the DTO with or without the patient present; and integration of PDQ responses into the DSC to ensure closed-looped communication with clinicians. Analysis of focus groups confirmed that tight integration of the interventions with the EHR would be necessary to prompt clinicians to reconsider the working diagnosis in cases with elevated diagnostic error (DE) risk or uncertainty. Potential implementation barriers included alert fatigue and distrust of the risk algorithm (DSC); time constraints, redundancies, and concerns about disclosing uncertainty to patients (DTO); and patient disagreement with the care team's diagnosis (PDQ).

Discussion

A user-centered approach led to evolution of requirements for 3 interventions targeting key diagnostic process failures in hospitalized patients at risk for DE.

Conclusions

We identify challenges and offer lessons from our user-centered design process.

Polygenic risk scores in pharmacogenomics: opportunities and challenges-a mini review

Pharmacogenomics (PGx) aims at tailoring drug therapy by considering patient genetic makeup. While drug dosage guidelines have been extensively based on single gene mutations (single nucleotide polymorphisms) over the last decade, polygenic risk scores (PRS) have emerged in the past years as a promising tool to account for the complex interplay and polygenic nature of patients' genetic predisposition affecting drug response. Even though PRS research has demonstrated convincing evidence in disease risk prediction, the clinical utility and its implementation in daily care has yet to be demonstrated, and pharmacogenomics is no exception; usual endpoints include drug efficacy or toxicity. Here, we review the general pipeline in PRS calculation, and we discuss some of the remaining barriers and challenges that must be undertaken to bring PRS research in PGx closer to patient care. Besides the need in following reporting guidelines and larger PGx patient cohorts, PRS integration will require close collaboration between bioinformatician, treating physicians and genetic consultants to ensure a transparent, generalizable, and trustful implementation of PRS results in real-world medical decisions.

Documentation dynamics: Note composition, burden, and physician efficiency

OBJECTIVE

To analyze how physician clinical note length and composition relate to electronic health record (EHR)-based measures of burden and efficiency that have been tied to burnout.

DATA SOURCES AND STUDY SETTING

Secondary EHR use metadata capturing physician-level measures from 203,728 US-based ambulatory physicians using the Epic Systems EHR between September 2020 and May 2021.

STUDY DESIGN

In this cross-sectional study, we analyzed physician clinical note length and note composition (e.g., content from manual or templated text). Our primary outcomes were three time-based measures of EHR burden (time writing EHR notes, time in the EHR after-hours, and EHR time on unscheduled days), and one measure of efficiency (percent of visits closed in the same day). We used multivariate regression to estimate the relationship between our outcomes and note length and composition.

DATA EXTRACTION

Physician-week measures of EHR usage were extracted from Epic's Signal platform used for measuring provider EHR efficiency. We calculated physician-level averages for our measures of interest and assigned physicians to overall note length deciles and note composition deciles from six sources, including templated text, manual text, and copy/paste text.

PRINCIPAL FINDINGS

Physicians in the top decile of note length demonstrated greater burden and lower efficiency than the median physician, spending 39% more time in the EHR after hours (p < 0.001) and closing 5.6 percentage points fewer visits on the same day (p < 0.001). Copy/paste demonstrated a similar dose/response relationship, with top-decile copy/paste users closing 6.8 percentage points fewer visits on the same day (p < 0.001) and spending more time in the EHR after hours and on days off (both p < 0.001). Templated text (e.g., Epic's SmartTools) demonstrated a non-linear relationship with burden and efficiency, with very low and very high levels of use associated with increased EHR burden and decreased efficiency.

CONCLUSIONS

"Efficiency tools" like copy/paste and templated text meant to reduce documentation burden and increase provider efficiency may have limited efficacy.

Zero-shot Interpretable Phenotyping of Postpartum Hemorrhage Using Large Language Models

Many areas of medicine would benefit from deeper, more accurate phenotyping, but there are limited approaches for phenotyping using clinical notes without substantial annotated data. Large language models (LLMs) have demonstrated immense potential to adapt to novel tasks with no additional training by specifying task-specific i nstructions. We investigated the per-formance of a publicly available LLM, Flan-T5, in phenotyping patients with postpartum hemorrhage (PPH) using discharge notes from electronic health records ( n =271,081). The language model achieved strong performance in extracting 24 granular concepts associated with PPH. Identifying these granular concepts accurately allowed the development of inter-pretable, complex phenotypes and subtypes. The Flan-T5 model achieved high fidelity in phenotyping PPH (positive predictive value of 0.95), identifying 47% more patients with this complication compared to the current standard of using claims codes. This LLM pipeline can be used reliably for subtyping PPH and outperformed a claims-based approach on the three most common PPH subtypes associated with uterine atony, abnormal placentation, and obstetric trauma. The advantage of this approach to subtyping is its interpretability, as each concept contributing to the subtype determination can be evaluated. Moreover, as definitions may change over time due to new guidelines, using granular concepts to create complex phenotypes enables prompt and efficient updating of the algorithm. Using this lan-guage modelling approach enables rapid phenotyping without the need for any manually annotated training data across multiple clinical use cases.

Correction: Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson

[This corrects the article DOI: 10.2196/43960.].

Refining Clinical Phenotypes to Improve Clinical Decision Support and Reduce Alert Fatigue: A Feasibility Study

BACKGROUND

Chronic kidney disease (CKD) is common and associated with adverse clinical outcomes. Most care for early CKD is provided in primary care, including hypertension (HTN) management. Computerized clinical decision support (CDS) can improve the quality of care for CKD but can also cause alert fatigue for primary care physicians (PCPs). Computable phenotypes (CPs) are algorithms to identify disease populations using, for example, specific laboratory data criteria.

OBJECTIVES

Our objective was to determine the feasibility of implementation of CDS alerts by developing CPs and estimating potential alert burden.

METHODS

We utilized clinical guidelines to develop a set of five CPs for patients with stage 3 to 4 CKD, uncontrolled HTN, and indications for initiation or titration of guideline-recommended antihypertensive agents. We then conducted an iterative data analytic process consisting of database queries, data validation, and subject matter expert discussion, to make iterative changes to the CPs. We estimated the potential alert burden to make final decisions about the scope of the CDS alerts. Specifically, the number of times that each alert could fire was limited to once per patient.

RESULTS

In our primary care network, there were 239,339 encounters for 105,992 primary care patients between April 1, 2018 and April 1, 2019. Of these patients, 9,081 (8.6%) had stage 3 and 4 CKD. Almost half of the CKD patients, 4,191 patients, also had uncontrolled HTN. The majority of CKD patients were female, elderly, white, and English-speaking. We estimated that 5,369 alerts would fire if alerts were triggered multiple times per patient, with a mean number of alerts shown to each PCP ranging from 0.07-to 0.17 alerts per week.

CONCLUSION

Development of CPs and estimation of alert burden allows researchers to iteratively fine-tune CDS prior to implementation. This method of assessment can help organizations balance the tradeoff between standardization of care and alert fatigue.

Racism and Electronic Health Records (EHRs): Perspectives for research and practice

Informatics researchers and practitioners have started exploring racism related to the implementation and use of electronic health records (EHRs). While this work has begun to expose structural racism which is a fundamental driver of racial and ethnic disparities, there is a lack of inclusion of concepts of racism in this work. This perspective provides a classification of racism at 3 levels-individual, organizational, and structural-and offers recommendations for future research, practice, and policy. Our recommendations include the need to capture and use structural measures of social determinants of health to address structural racism, intersectionality as a theoretical framework for research, structural competency training, research on the role of prejudice and stereotyping in stigmatizing documentation in EHRs, and actions to increase the diversity of private sector informatics workforce and participation of minority scholars in specialty groups. Informaticians have an ethical and moral obligation to address racism, and private and public sector organizations have a transformative role in addressing equity and racism associated with EHR implementation and use.

Digital healthcare equity in primary care: implementing an integrated digital health navigator

The 21st Century Cures Act and the rise of telemedicine led to renewed focus on patient portals. However, portal use disparities persist and are in part driven by limited digital literacy. To address digital disparities in primary care, we implemented an integrated digital health navigator program supporting portal use among patients with type II diabetes. During our pilot, we were able to enroll 121 (30.9%) patients onto the portal. Of newly enrolled or trained patients, 75 (62.0%) were Black, 13 (10.7%) were White, 23 (19.0%) were Hispanic/Latinx, 4 (3.3%) were Asian, 3 (2.5%) were of another race or ethnicity, and 3 (2.5%) had missing data. Our overall portal enrollment for clinic patients with type II diabetes increased for Hispanic/Latinx patients from 30% to 42% and Black patients from 49% to 61%. We used the Consolidated Framework for Implementation Research to understand key implementation components. Using our approach, other clinics can implement an integrated digital health navigator to support patient portal use.

Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson

BACKGROUND

Evidence-based point-of-care information (POCI) tools can facilitate patient safety and care by helping clinicians to answer disease state and drug information questions in less time and with less effort. However, these tools may also be visually challenging to navigate or lack the comprehensiveness needed to sufficiently address a medical issue.

OBJECTIVE

This study aimed to collect clinicians' feedback and directly observe their use of the combined POCI tool DynaMed and Micromedex with Watson, now known as DynaMedex. EBSCO partnered with IBM Watson Health, now known as Merative, to develop the combined tool as a resource for clinicians. We aimed to identify areas for refinement based on participant feedback and examine participant perceptions to inform further development.

METHODS

Participants (N=43) within varying clinical roles and specialties were recruited from Brigham and Women's Hospital and Massachusetts General Hospital in Boston, Massachusetts, United States, between August 10, 2021, and December 16, 2021, to take part in usability sessions aimed at evaluating the efficiency and effectiveness of, as well as satisfaction with, the DynaMed and Micromedex with Watson tool. Usability testing methods, including think aloud and observations of user behavior, were used to identify challenges regarding the combined tool. Data collection included measurements of time on task; task ease; satisfaction with the answer; posttest feedback on likes, dislikes, and perceived reliability of the tool; and interest in recommending the tool to a colleague.

RESULTS

On a 7-point Likert scale, pharmacists rated ease (mean 5.98, SD 1.38) and satisfaction (mean 6.31, SD 1.34) with the combined POCI tool higher than the physicians, nurse practitioner, and physician's assistants (ease: mean 5.57, SD 1.64, and satisfaction: mean 5.82, SD 1.60). Pharmacists spent longer (mean 2 minutes, 26 seconds, SD 1 minute, 41 seconds) on average finding an answer to their question than the physicians, nurse practitioner, and physician's assistants (mean 1 minute, 40 seconds, SD 1 minute, 23 seconds).

CONCLUSIONS

Overall, the tool performed well, but this usability evaluation identified multiple opportunities for improvement that would help inexperienced users.

Physician Note Composition Patterns and Time on the EHR Across Specialty Types: a National, Cross-sectional Study

BACKGROUND

The burden of clinical documentation in electronic health records (EHRs) has been associated with physician burnout. Numerous tools (e.g., note templates and dictation services) exist to ease documentation burden, but little evidence exists regarding how physicians use these tools in combination and the degree to which these strategies correlate with reduced time spent on documentation.

OBJECTIVE

To characterize EHR note composition strategies, how these strategies differ in time spent on notes and the EHR, and their distribution across specialty types.

DESIGN

Secondary analysis of physician-level measures of note composition and EHR use derived from Epic Systems' Signal data warehouse. We used k-means clustering to identify documentation strategies, and ordinary least squares regression to analyze the relationship between documentation strategies and physician time spent in the EHR, on notes, and outside scheduled hours.

PARTICIPANTS

A total of 215,207 US-based ambulatory physicians using the Epic EHR between September 2020 and May 2021.

MAIN MEASURES

Percent of note text derived from each of five documentation tools: SmartTools, copy/paste, manual text, NoteWriter, and voice recognition and transcription; average total and after-hours EHR time per visit; average time on notes per visit.

KEY RESULTS

Six distinct note composition strategies emerged in cluster analyses. The most common strategy was predominant SmartTools use (n=89,718). In adjusted analyses, physicians using primarily transcription and dictation (n=15,928) spent less time on notes than physicians with predominant Smart Tool use. (b=-1.30, 95% CI=-1.62, -0.99, p<0.001; average 4.8 min per visit), while those using mostly copy/paste (n=23,426) spent more time on notes (b=2.38, 95% CI=1.92, 2.84, p<0.001; average 13.1 min per visit).

CONCLUSIONS

Physicians' note composition strategies have implications for both time in notes and after-hours EHR use, suggesting that how physicians use EHR-based documentation tools can be a key lever for institutions investing in EHR tools and training to reduce documentation time and alleviate EHR-associated burden.

Trend and Co-occurrence Network of COVID-19 Symptoms From Large-Scale Social Media Data: Infoveillance Study

BACKGROUND

For an emergent pandemic, such as COVID-19, the statistics of symptoms based on hospital data may be biased or delayed due to the high proportion of asymptomatic or mild-symptom infections that are not recorded in hospitals. Meanwhile, the difficulty in accessing large-scale clinical data also limits many researchers from conducting timely research.

OBJECTIVE

Given the wide coverage and promptness of social media, this study aimed to present an efficient workflow to track and visualize the dynamic characteristics and co-occurrence of symptoms for the COVID-19 pandemic from large-scale and long-term social media data.

METHODS

This retrospective study included 471,553,966 COVID-19-related tweets from February 1, 2020, to April 30, 2022. We curated a hierarchical symptom lexicon for social media containing 10 affected organs/systems, 257 symptoms, and 1808 synonyms. The dynamic characteristics of COVID-19 symptoms over time were analyzed from the perspectives of weekly new cases, overall distribution, and temporal prevalence of reported symptoms. The symptom evolutions between virus strains (Delta and Omicron) were investigated by comparing the symptom prevalence during their dominant periods. A co-occurrence symptom network was developed and visualized to investigate inner relationships among symptoms and affected body systems.

RESULTS

This study identified 201 COVID-19 symptoms and grouped them into 10 affected body systems. There was a significant correlation between the weekly quantity of self-reported symptoms and new COVID-19 infections (Pearson correlation coefficient=0.8528; P<.001). We also observed a 1-week leading trend (Pearson correlation coefficient=0.8802; P<.001) between them. The frequency of symptoms showed dynamic changes as the pandemic progressed, from typical respiratory symptoms in the early stage to more musculoskeletal and nervous symptoms in the later stages. We identified the difference in symptoms between the Delta and Omicron periods. There were fewer severe symptoms (coma and dyspnea), more flu-like symptoms (throat pain and nasal congestion), and fewer typical COVID symptoms (anosmia and taste altered) in the Omicron period than in the Delta period (all P<.001). Network analysis revealed co-occurrences among symptoms and systems corresponding to specific disease progressions, including palpitations (cardiovascular) and dyspnea (respiratory), and alopecia (musculoskeletal) and impotence (reproductive).

CONCLUSIONS

This study identified more and milder COVID-19 symptoms than clinical research and characterized the dynamic symptom evolution based on 400 million tweets over 27 months. The symptom network revealed potential comorbidity risk and prognostic disease progression. These findings demonstrate that the cooperation of social media and a well-designed workflow can depict a holistic picture of pandemic symptoms to complement clinical studies.

Association of Clinician Practice Ownership With Ability of Primary Care Practices to Improve Quality Without Increasing Burnout

Importance

Work environments and practice structural features are associated with both burnout and the ability of practices to enhance quality of care.

Objective

To characterize factors associated with primary care practices successfully improving quality scores without increasing clinician and staff burnout.

Design, Setting, and Participants

This cross-sectional study assessed small- to medium-sized primary care practices that participated in the EvidenceNOW: Advancing Heart Health initiative using surveys that were administered at baseline (September 2015 to April 2017) and after the intervention (January 2017 to October 2018). Data were analyzed from February 2022 to January 2023.

Main Outcomes and Measures

The primary outcome of being a quality and well-being positive deviant practice was defined as a practice with a stable or improved percentage of clinicians and staff reporting burnout over the study period and with practice-level improvement in all 3 cardiovascular quality measures: aspirin prescribing, blood pressure control, and smoking cessation counseling.

Results

Of 727 practices with complete burnout and aspirin prescribing, blood pressure control, and smoking cessation counseling data, 18.3% (n = 133) met the criteria to be considered quality and well-being positive deviant practices. In analyses adjusted for practice location, accountable care organization and demonstration project participation, and practice specialty composition, clinician-owned practices had greater odds of being a positive deviant practice (odds ratio, 2.02; 95% CI, 1.16-3.54) than practices owned by a hospital or health system.

Conclusions and Relevance

In this cross-sectional study, clinician-owned practices were more likely to achieve improvements in cardiovascular quality outcomes without increasing staff member burnout than were practices owned by a hospital or health system. Given increasing health care consolidation, our findings suggest the value of studying cultural features of clinician-owned practices that may be associated with positive quality and experience outcomes.