Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study

AI-enabled electrocardiography alert intervention and all-cause mortality: a pragmatic randomized clinical trial

The early identification of vulnerable patients has the potential to improve outcomes but poses a substantial challenge in clinical practice. This study evaluated the ability of an artificial intelligence (AI)-enabled electrocardiogram (ECG) to identify hospitalized patients with a high risk of mortality in a multisite randomized controlled trial involving 39 physicians and 15,965 patients. The AI-ECG alert intervention included an AI report and warning messages delivered to the physicians, flagging patients predicted to be at high risk of mortality. The trial met its primary outcome, finding that implementation of the AI-ECG alert was associated with a significant reduction in all-cause mortality within 90 days: 3.6% patients in the intervention group died within 90 days, compared to 4.3% in the control group (4.3%) (hazard ratio (HR) = 0.83, 95% confidence interval (CI) = 0.70-0.99). A prespecified analysis showed that reduction in all-cause mortality associated with the AI-ECG alert was observed primarily in patients with high-risk ECGs (HR = 0.69, 95% CI = 0.53-0.90). In analyses of secondary outcomes, patients in the intervention group with high-risk ECGs received increased levels of intensive care compared to the control group; for the high-risk ECG group of patients, implementation of the AI-ECG alert was associated with a significant reduction in the risk of cardiac death (0.2% in the intervention arm versus 2.4% in the control arm, HR = 0.07, 95% CI = 0.01-0.56). While the precise means by which implementation of the AI-ECG alert led to decreased mortality are to be fully elucidated, these results indicate that such implementation assists in the detection of high-risk patients, prompting timely clinical care and reducing mortality. ClinicalTrials.gov registration: NCT05118035 .

Characterization of pentosan polysulfate patients for development of an alert and screening system for ophthalmic monitoring

OBJECTIVE

Pentosan polysulfate (PPS; ELMIRON, Janssen Pharmaceuticals, Titusville, NJ) is a U.S. Food and Drug Administration-approved oral medication for interstitial cystitis. Numerous reports have been published detailing retinal toxicity with the use of PPS. Studies characterizing this condition are primarily retrospective, and consequently, alert and screening systems need to be developed to actively screen for this disease. The goal of this study was to characterize ophthalmic monitoring trends of a PPS-using patient sample to construct an alert and screening system for monitoring this condition.

METHODS

A single-institution retrospective chart review was conducted between January 2005 and November 2020 to characterize PPS use. An electronic medical record (EMR) alert was constructed to trigger based on new PPS prescriptions and renewals offering ophthalmology referral.

RESULTS

A total of 1407 PPS users over 15 years was available for characterization, with 1220 (86.7%) being female, the average duration of exposure being 71.2 ± 62.6 months, and the average medication cumulative exposure being 669.7 ± 569.2 g. A total of 151 patients (10.7%) had a recorded visit with an ophthalmologist, with 71 patients (5.0%) having optical coherence tomography imaging. The EMR alert fired for 88 patients over 1 year, with 34 patients (38.6%) either already being screened by an ophthalmologist or having been referred for screening.

CONCLUSIONS

An EMR support tool can improve referral rates of PPS maculopathy screening with an ophthalmologist and may serve as an efficient method for longitudinal screening of this condition with the added benefit of informing pentosan polysulfate prescribers about this condition. Effective screening and detection may help determine which patients are at high risk for this condition.

Resident-Driven Clinical Decision Support Governance to Improve the Utility of Clinical Decision Support

OBJECTIVES

 This resident-driven quality improvement project aimed to better understand the known problem of a misaligned clinical decision support (CDS) strategy and improve CDS utilization.

METHODS

 An internal survey was sent to all internal medicine (IM) residents to identify the most bothersome CDS alerts. Survey results were supported by electronic health record (EHR) data of CDS firing rates and response rates which were collected for each of the three most bothersome CDS tools. Changes to firing criteria were created to increase utilization and to better align with the five rights of CDS. Findings and proposed changes were presented to our institution's CDS Governance Committee. Changes were approved and implemented. Postintervention firing rates were then collected for 1 week.

RESULTS

 Twenty nine residents participated in the CDS survey and identified sepsis alerts, lipid profile reminders, and telemetry renewals to be the most bothersome alerts. EHR data showed action rates for these CDS as low as 1%. We implemented changes to focus emergency department (ED)-based sepsis alerts to the right provider, better address the right information for lipid profile reminders, and select the right time in workflow for telemetry renewals to be most effective. With these changes we successfully eliminated ED-based sepsis CDS reminders for IM providers, saw a 97% reduction in firing rates for the lipid profile CDS, and noted a 55% reduction in firing rates for telemetry CDS.

CONCLUSION

 This project highlighted that alert improvements spearheaded by resident teams can be completed successfully using robust CDS governance strategies and can effectively optimize interruptive alerts.

Randomized Comparison of Electronic Health Record Alert Types in Eliciting Responses about Prognosis in Gynecologic Oncology Patients

OBJECTIVES

 To compare the ability of different electronic health record alert types to elicit responses from users caring for cancer patients benefiting from goals of care (GOC) conversations.

METHODS

 A validated question asking if the user would be surprised by the patient's 6-month mortality was built as an Epic BestPractice Advisory (BPA) alert in three versions-(1) Required on Open chart (pop-up BPA), (2) Required on Close chart (navigator BPA), and (3) Optional Persistent (Storyboard BPA)-randomized using patient medical record number. Meaningful responses were defined as "Yes" or "No," rather than deferral. Data were extracted over 6 months.

RESULTS

 Alerts appeared for 685 patients during 1,786 outpatient encounters. Measuring encounters where a meaningful response was elicited, rates were highest for Required on Open (94.8% of encounters), compared with Required on Close (90.1%) and Optional Persistent (19.7%) (p < 0.001). Measuring individual alerts to which responses were given, they were most likely meaningful with Optional Persistent (98.3% of responses) and least likely with Required on Open (68.0%) (p < 0.001). Responses of "No," suggesting poor prognosis and prompting GOC, were more likely with Optional Persistent (13.6%) and Required on Open (10.3%) than with Required on Close (7.0%) (p = 0.028).

CONCLUSION

 Required alerts had response rates almost five times higher than optional alerts. Timing of alerts affects rates of meaningful responses and possibly the response itself. The alert with the most meaningful responses was also associated with the most interruptions and deferral responses. Considering tradeoffs in these metrics is important in designing clinical decision support to maximize success.

Associations of perceived changes in work due to digitalization and the amount of digital work with job strain among physicians: a national representative sample

BACKGROUND

Physicians' work is often stressful. The digitalization of healthcare aims to streamline work, but not all physicians have experienced its realization. We examined associations of perceived changes in work due to digitalization and the amount of digital work with job strain among physicians. The moderating role of the length of work experience was investigated for these associations.

METHODS

We used representative survey data on Finnish physicians' (N = 4271) experiences of digitalization from 2021. The independent variables included perceptions on statements about work transformations aligned with digitalization goals, and the extent that information systems and teleconsultations were utilized. Stress related to information systems (SRIS), time pressure, and psychological stress were the dependent variables. We analyzed the associations using multivariable linear and logistic regressions.

RESULTS

Respondents had a mean SRIS score of 3.5 and a mean time pressure score of 3.7 on a scale of 1-5. Psychological stress was experienced by 60%. Perceptions associated with higher SRIS comprised disagreements with statements asserting that digitalization accelerates clinical encounters (b = .23 [95% CI: .16-.30]), facilitates access to patient information (b = .15 [.07-.23]), and supports decision-making (b = .11 [.05-.18]). Disagreement with accelerated clinical encounters (b = .12 [.04-.20]), and agreements with patients' more active role in care (b = .11 [.04-.19]) and interprofessional collaboration (b = .10 [.02-.18]) were opinions associated with greater time pressure. Disagreeing with supported decision-making (OR = 1.26 [1.06-1.48]) and agreeing with patients' active role (OR = 1.19 [1.02-1.40]) were associated with greater psychological stress. However, perceiving improvements in the pace of clinical encounters and access to patient information appeared to alleviate job strain. Additionally, extensive digital work was consistently linked to higher strain. Those respondents who held teleconsultations frequently and had less than 6 years of work experience reported the greatest levels of time pressure.

CONCLUSIONS

Physicians seem to be strained by frequent teleconsultations and work that does not meet the goals of digitalization. Improving physicians' satisfaction with digitalization through training specific to the stage of career and system development can be crucial for their well-being. Schedules for digital tasks should be planned and allocated to prevent strain related to achieving the digitalization goals.

A Data-Driven Approach to Evaluate Barcode-Assisted Medication Preparation Alerts at a Large Academic Medical Center

BACKGROUND

The purpose of this study was to develop a data-driven process to analyze barcode-assisted medication preparation alert data with a goal of minimizing inaccurate alerts.

METHODS

Medication preparation data for the prior three-month period was obtained from an electronic health record system. A dashboard was developed to identify recurrent, high-volume alerts and associated medication records. A randomization tool was used to obtain a prespecified proportion of the alerts to review for appropriateness. Alert root causes were identified by chart review. Depending on the alert's cause(s), targeted informatics build changes, workflow and purchasing changes, and/or staff education were implemented. The rate of alerts was measured postintervention for select drugs.

RESULTS

The institution averaged 31,000 medication preparation alerts per month. The "barcode not recognized" alert (13,000) was the highest volume over the study period. Eighty-five medication records were identified as contributing to a high volume of alerts (5,200/31,000), representing 49 unique drugs. Of the 85 medication records triggering alerts, 36 required staff education, 22 required informatics build changes, and 8 required workflow changes. Targeted interventions for 2 medications, resulted in reducing the rate of the "barcode not recognized" alert from 26.6% to 1.3% for polyethylene glycol and from 48.7% to 0% for cyproheptadine.

CONCLUSION

This quality improvement project highlighted opportunities to improve medication purchasing, storage, and preparation through development of a standard process to evaluate barcode-assisted medication preparation alert data. A data-driven approach can help identify and minimize inaccurate alerts ("noise") and promote medication safety.

Beyond the override: Using evidence of previous drug tolerance to suppress drug allergy alerts; a retrospective study of opioid alerts

OBJECTIVE

Despite the extensive literature exploring alert fatigue, most studies have focused on describing the phenomenon, but not on fixing it. The authors aimed to identify data useful to avert clinically irrelevant alerts to inform future research on clinical decision support (CDS) design.

METHODS

We conducted a retrospective observational study of opioid drug allergy alert (DAA) overrides for the calendar year of 2019 at a large academic medical center, to identify data elements useful to find irrelevant alerts to be averted.

RESULTS

Overall, 227,815 DAAs were fired in 2019, with an override rate of 91 % (n = 208196). Opioids represented nearly two-thirds of these overrides (n = 129063; 62 %) and were the drug class with the highest override rate (96 %). On average, 29 opioid DAAs were overridden per patient. While most opioid alerts (97.1 %) are fired for a possible match (the drug class of the allergen matches the drug class of the prescribed drug), they are overridden significantly less frequently for definite match (exact match between allergen and prescribed drug) (88 % vs. 95.9 %, p < 0.001). When comparing the triggering drug with previously administered drugs, override rates were equally high for both definite match (95.9 %), no match (95.5 %), and possible match (95.1 %). Likewise, when comparing to home medications, overrides were excessively high for possible match (96.3 %), no match (96 %), and definite match (94.4 %).

CONCLUSION

We estimate that 74.5% of opioid DAAs (46.4% of all DAAs) at our institution could be relatively safely averted, since they either have a definite match for previous inpatient administrations suggesting drug tolerance or are fired as possible match with low risk of cross-sensitivity. Future research should focus on identifying other relevant data elements ideally with automated methods and use of emerging standards to empower CDS systems to suppress false-positive alerts while avoiding safety hazards.

Qualitative analysis of a remote monitoring intervention for managing heart failure

BACKGROUND

Heart failure (HF) is one of the most common reasons for hospital admission and is a major cause of morbidity, mortality, and increasing health care costs. The EMPOWER study was a randomized trial that used remote monitoring technology to track patients' weight and diuretic adherence and a state-of-the-art approach derived from behavioral economics to motivate adherence to the reverse monitoring technology.

OBJECTIVE

The goal was to explore patient and clinician perceptions of the program and its impact on perceived health outcomes and better understand why some patients or clinicians did better or worse than others in response to the intervention.

APPROACH

This was a retrospective qualitative study utilizing semi-structured interviews with 43 patients and 16 clinicians to understand the trial's processes, reflecting on successes and areas for improvement for future iterations of behavioral economic interventions.

KEY RESULTS

Many patients felt supported, and they appreciated the intervention. Many also appreciated the lottery intervention, and while it was not an incentive for enrolling for many respondents, it may have increased adherence during the study. Clinicians felt that the intervention integrated well into their workflow, but the number of alerts was burdensome. Additionally, responses to alerts varied considerably by provider, perhaps because there are no professional guidelines for alerts unaccompanied by severe symptoms.

CONCLUSION

Our qualitative analysis indicates potential areas for additional exploration and consideration to design better behavioral economic interventions to improve cardiovascular health outcomes for patients with HF. Patients appreciated lottery incentives for adhering to program requirements; however, many were too far along in their disease progression to benefit from the intervention. Clinicians found the amount and frequency of electronic alerts burdensome and felt they did not improve patient outcomes.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02708654.

Mixed-methods Evaluation of an Expedited Partner Therapy Take-home Medication Program: Pilot Emergency Department Intervention to Improve Sexual Health Equity

Background: Treatment for partners of patients diagnosed with sexually transmitted infections (STI), referred to as expedited partner therapy (EPT), is infrequently used in the emergency department (ED). This was a pilot program to initiate and evaluate EPT through medication-in-hand ("take-home") kits or paper prescriptions. In this study we aimed to assess the frequency of EPT prescribing, the efficacy of a randomized best practice advisory (BPA) on the uptake, perceptions of emergency clinicians regarding the EPT pilot, and factors associated with EPT prescribing. Methods: We conducted this pilot study at an academic ED in the midwestern US between August-October 2021. The primary outcome of EPT prescription uptake and the BPA impact was measured via chart abstraction and analyzed through summary statistics and the Fisher exact test. We analyzed the secondary outcome of barriers and facilitators to program implementation through ED staff interviews (physicians, physician assistants, and nurses). We used a rapid qualitative assessment method for the analysis of the interviews. Results: During the study period, 52 ED patients were treated for chlamydia/gonorrhea, and EPT was offered to 25% (95% CI 15%-39%) of them. Expedited partner therapy was prescribed significantly more often (42% vs 8%; P < 0.01) when the interruptive pop-up alert BPA was shown compared to not shown. Barriers identified in the interviews included workflow constraints and knowledge of EPT availability. The BPA was viewed positively by the majority of participants. Conclusion: In this pilot EPT program, expedited partner therapy was provided to 25% of ED patients who appeared eligible to receive it. The interruptive pop-up alert BPA significantly increased EPT prescribing. Barriers identified to EPT prescribing should be the subject of future interventions to improve provision of EPT from the emergency department.

User-Centered Design and Evaluation of Clinical Decision Support to Improve Early Peanut Introduction: Formative Study

BACKGROUND

Peanut allergy has recently become more prevalent. Peanut introduction recommendations have evolved from suggesting peanut avoidance until the age of 3 years to more recent guidelines encouraging early peanut introduction after the Learning Early about Peanut Allergy (LEAP) study in 2015. Guideline adherence is poor, leading to missed care opportunities.

OBJECTIVE

In this study, we aimed to develop a user-centered clinical decision support (CDS) tool to improve implementation of the most recent early peanut introduction guidelines in the primary care clinic setting.

METHODS

We edited the note template of the well-child check (WCC) visits at ages 4 and 6 months with CDS prompts and point-of-care education. Formative and summative usability testing were completed with pediatric residents in a simulated electronic health record (EHR). We estimated task completion rates and perceived usefulness of the CDS in summative testing, comparing a test EHR with and without the CDS.

RESULTS

Formative usability testing with the residents provided qualitative data that led to improvements in the build for both the 4-month and 6-month WCC note templates. During summative usability testing, the CDS tool significantly improved discussion of early peanut introduction at the 4-month WCC visit compared to scenarios without the CDS tool (9/15, 60% with CDS and 0/15, 0% without CDS). All providers except one at the 4-month WCC scenario gave at least an adequate score for the ease of use of the CDS tool for the history of present illness and assessment and plan sections. During the summative usability testing with the 6-month WCC new build note template, providers more commonly provided comprehensive care once obtaining a patient history concerning for an immunoglobulin E-mediated peanut reaction by placing a referral to allergy/immunology (P=.48), prescribing an epinephrine auto-injector (P=.07), instructing on how to avoid peanut products (P<.001), and providing an emergency treatment plan (P=.003) with CDS guidance. All providers gave at least an adequate score for ease of use of the CDS tool in the after-visit summary.

CONCLUSIONS

User-centered CDS improved application of early peanut introduction recommendations and comprehensive care for patients who have symptoms concerning for peanut allergy in a simulation.

Cognitive perspectives on maintaining physicians' medical expertise: I. Reimagining Maintenance of Certification to promote lifelong learning

Until recently, physicians in the USA who were board-certified in a specialty needed to take a summative test every 6-10 years. However, the 24 Member Boards of the American Board of Medical Specialties are in the process of switching toward much more frequent assessments, which we refer to as longitudinal assessment. The goal of longitudinal assessments is to provide formative feedback to physicians to help them learn content they do not know as well as serve an evaluation for board certification. We present five articles collectively covering the science behind this change, the likely outcomes, and some open questions. This initial article introduces the context behind this change. This article also discusses various forms of lifelong learning opportunities that can help physicians stay current, including longitudinal assessment, and the pros and cons of each.

Epic Sepsis Model Inpatient Predictive Analytic Tool: A Validation Study

Earlier treatment of sepsis leads to decreased mortality. Epic is an electronic medical record providing a predictive alert system for sepsis, the Epic Sepsis Model (ESM) Inpatient Predictive Analytic Tool. External validation of this system is lacking. This study aims to evaluate the ESM as a sepsis screening tool and determine whether an association exists between ESM alert system implementation and subsequent sepsis-related mortality.

DESIGN

Before-and-after study comparing baseline and intervention period.

SETTING

Urban 746-bed academic level 1 trauma center.

PATIENTS

Adult acute care inpatients discharged between January 12, 2018, and July 31, 2019.

INTERVENTIONS

During the before period, ESM was turned on in the background, but nurses and providers were not alerted of results. The system was then activated to alert providers of scores greater than or equal to 5, a set point determined using receiver operating characteristic curve analysis (area under the curve, 0.834; p < 0.001).

MEASUREMENTS AND MAIN RESULTS

Primary outcome was mortality during hospitalization; secondary outcomes were sepsis order set utilization, length of stay, and timing of administration of sepsis-appropriate antibiotics. Of the 11,512 inpatient encounters assessed by ESM, 10.2% (1,171) had sepsis based on diagnosis codes. As a screening test, the ESM had sensitivity, specificity, positive predictive value, and negative predictive value rates of 86.0%, 80.8%, 33.8%, and 98.11%, respectively. After ESM implementation, unadjusted mortality rates in patients with ESM score greater than or equal to 5 and who had not yet received sepsis-appropriate antibiotics declined from 24.3% to 15.9%; multivariable analysis yielded an odds ratio of sepsis-related mortality (95% CI) of 0.56 (0.39-0.80).

CONCLUSIONS

In this single-center before-and-after study, utilization of the ESM score as a screening test was associated with a 44% reduction in the odds of sepsis-related mortality. Due to wide utilization of Epic, this is a potentially promising tool to improve sepsis mortality in the United States. This study is hypothesis generating, and further work with more rigorous study design is needed.

Leveraging electronic health records to streamline the conduct of cardiovascular clinical trials

Conventional randomized controlled trials (RCTs) can be expensive, time intensive, and complex to conduct. Trial recruitment, participation, and data collection can burden participants and research personnel. In the past two decades, there have been rapid technological advances and an exponential growth in digitized healthcare data. Embedding RCTs, including cardiovascular outcome trials, into electronic health record systems or registries may streamline screening, consent, randomization, follow-up visits, and outcome adjudication. Moreover, wearable sensors (i.e. health and fitness trackers) provide an opportunity to collect data on cardiovascular health and risk factors in unprecedented detail and scale, while growing internet connectivity supports the collection of patient-reported outcomes. There is a pressing need to develop robust mechanisms that facilitate data capture from diverse databases and guidance to standardize data definitions. Importantly, the data collection infrastructure should be reusable to support multiple cardiovascular RCTs over time. Systems, processes, and policies will need to have sufficient flexibility to allow interoperability between different sources of data acquisition. Clinical research guidelines, ethics oversight, and regulatory requirements also need to evolve. This review highlights recent progress towards the use of routinely generated data to conduct RCTs and discusses potential solutions for ongoing barriers. There is a particular focus on methods to utilize routinely generated data for trials while complying with regional data protection laws. The discussion is supported with examples of cardiovascular outcome trials that have successfully leveraged the electronic health record, web-enabled devices or administrative databases to conduct randomized trials.

Provider Interaction With an Electronic Health Record Notification to Identify Eligible Patients for a Cluster Randomized Trial of Advance Care Planning in Primary Care: Secondary Analysis

BACKGROUND

Advance care planning (ACP) improves patient-provider communication and aligns care to patient values, preferences, and goals. Within a multisite Meta-network Learning and Research Center ACP study, one health system deployed an electronic health record (EHR) notification and algorithm to alert providers about patients potentially appropriate for ACP and the clinical study.

OBJECTIVE

The aim of the study is to describe the implementation and usage of an EHR notification for referring patients to an ACP study, evaluate the association of notifications with study referrals and engagement in ACP, and assess provider interactions with and perspectives on the notifications.

METHODS

A secondary analysis assessed provider usage and their response to the notification (eg, acknowledge, dismiss, or engage patient in ACP conversation and refer patient to the clinical study). We evaluated all patients identified by the EHR algorithm during the Meta-network Learning and Research Center ACP study. Descriptive statistics compared patients referred to the study to those who were not referred to the study. Health care utilization, hospice referrals, and mortality as well as documentation and billing for ACP and related legal documents are reported. We evaluated associations between notifications with provider actions (ie, referral to study, ACP not documentation, and ACP billing). Provider free-text comments in the notifications were summarized qualitatively. Providers were surveyed on their satisfaction with the notification.

RESULTS

Among the 2877 patients identified by the EHR algorithm over 20 months, 17,047 unique notifications were presented to 45 providers in 6 clinics, who then referred 290 (10%) patients. Providers had a median of 269 (IQR 65-552) total notifications, and patients had a median of 4 (IQR 2-8). Patients with more (over 5) notifications were less likely to be referred to the study than those with fewer notifications (57/1092, 5.2% vs 233/1785, 13.1%; P<.001). The most common free-text comment on the notification was lack of time. Providers who referred patients to the study were more likely to document ACP and submit ACP billing codes (P<.001). In the survey, 11 providers would recommend the notification (n=7, 64%); however, the notification impacted clinical workflow (n=9, 82%) and was difficult to navigate (n=6, 55%).

CONCLUSIONS

An EHR notification can be implemented to remind providers to both perform ACP conversations and refer patients to a clinical study. There were diminishing returns after the fifth EHR notification where additional notifications did not lead to more trial referrals, ACP documentation, or ACP billing. Creation and optimization of EHR notifications for study referrals and ACP should consider the provider user, their workflow, and alert fatigue to improve implementation and adoption.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03577002; https://clinicaltrials.gov/ct2/show/NCT03577002.

Refining Clinical Phenotypes to Improve Clinical Decision Support and Reduce Alert Fatigue: A Feasibility Study

BACKGROUND

Chronic kidney disease (CKD) is common and associated with adverse clinical outcomes. Most care for early CKD is provided in primary care, including hypertension (HTN) management. Computerized clinical decision support (CDS) can improve the quality of care for CKD but can also cause alert fatigue for primary care physicians (PCPs). Computable phenotypes (CPs) are algorithms to identify disease populations using, for example, specific laboratory data criteria.

OBJECTIVES

Our objective was to determine the feasibility of implementation of CDS alerts by developing CPs and estimating potential alert burden.

METHODS

We utilized clinical guidelines to develop a set of five CPs for patients with stage 3 to 4 CKD, uncontrolled HTN, and indications for initiation or titration of guideline-recommended antihypertensive agents. We then conducted an iterative data analytic process consisting of database queries, data validation, and subject matter expert discussion, to make iterative changes to the CPs. We estimated the potential alert burden to make final decisions about the scope of the CDS alerts. Specifically, the number of times that each alert could fire was limited to once per patient.

RESULTS

In our primary care network, there were 239,339 encounters for 105,992 primary care patients between April 1, 2018 and April 1, 2019. Of these patients, 9,081 (8.6%) had stage 3 and 4 CKD. Almost half of the CKD patients, 4,191 patients, also had uncontrolled HTN. The majority of CKD patients were female, elderly, white, and English-speaking. We estimated that 5,369 alerts would fire if alerts were triggered multiple times per patient, with a mean number of alerts shown to each PCP ranging from 0.07-to 0.17 alerts per week.

CONCLUSION

Development of CPs and estimation of alert burden allows researchers to iteratively fine-tune CDS prior to implementation. This method of assessment can help organizations balance the tradeoff between standardization of care and alert fatigue.

Research recruitment through the patient portal: perspectives of community focus groups in Seattle and Atlanta

Objective

Research recruitment through patient portals (ie, patient-facing, web-based clinical interfaces) has the potential to be effective, efficient, and inclusive, but best practices remain undefined. We sought to better understand how patients view this recruitment approach.

Materials and Methods

We conducted 6 focus groups in Atlanta, GA and Seattle, WA with members of patient advisory committees and the general public. Discussions addressed acceptability of patient portal recruitment and communication preferences. Focus groups were audio-recorded, transcribed, and analyzed using deductive and inductive codes. Iterative team discussions identified major themes.

Results

Of 49 total participants, 20 were patient advisory committee members. Participants' mean age was 49 (range 18-74); 59% identified as non-Hispanic White and 31% as Black/African American. Participants were supportive of patient portal recruitment and confident that messages were private and legitimate. Participants identified transparency and patient control over whether and how to participate as essential features. Concerns included the frequency of research messages and the ability to distinguish between research and clinical messages. Participants also discussed how patient portal recruitment might affect diversity and inclusion.

Discussion

Focus group participants generally found patient portal recruitment acceptable and perceived it as secure and trustworthy. Transparency, control, and attention to inclusiveness were identified as key considerations for developing best practices.

Conclusion

For institutions implementing patient portal recruitment programs, continued engagement with patient populations can help facilitate translation of these findings into best practices and ensure that implemented strategies accomplish intended goals.

Clinical decision support to Optimize Care of patients with Atrial Fibrillation or flutter in the Emergency department: protocol of a stepped-wedge cluster randomized pragmatic trial (O'CAFÉ trial)

BACKGROUND

Management of adults with atrial fibrillation (AF) or atrial flutter in the emergency department (ED) includes rate reduction, cardioversion, and stroke prevention. Different approaches to these components of care may lead to variation in frequency of hospitalization and stroke prevention actions, with significant implications for patient experience, cost of care, and risk of complications. Standardization using evidence-based recommendations could reduce variation in management, preventable hospitalizations, and stroke risk.

METHODS

We describe the rationale for our ED-based AF treatment recommendations. We also describe the development of an electronic clinical decision support system (CDSS) to deliver these recommendations to emergency physicians at the point of care. We implemented the CDSS at three pilot sites to assess feasibility and solicit user feedback. We will evaluate the impact of the CDSS on hospitalization and stroke prevention actions using a stepped-wedge cluster randomized pragmatic clinical trial across 13 community EDs in Northern California.

DISCUSSION

We hypothesize that the CDSS intervention will reduce hospitalization of adults with isolated AF or atrial flutter presenting to the ED and increase anticoagulation prescription in eligible patients at the time of ED discharge and within 30 days. If our hypotheses are confirmed, the treatment protocol and CDSS could be recommended to other EDs to improve management of adults with AF or atrial flutter.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05009225 .  Registered on 17 August 2021.

Electronic Health Record-Nested Reminders for Serum Lithium Level Monitoring in Patients With Mood Disorder: Randomized Controlled Trial

BACKGROUND

Clinical guidelines recommend regular serum lithium monitoring every 3 to 6 months. However, in the real world, only a minority of patients receive adequate monitoring.

OBJECTIVE

This study aims to examine whether the use of the electronic health record (EHR)-nested reminder system for serum lithium monitoring can help achieve serum lithium concentrations within the therapeutic range for patients on lithium maintenance therapy.

METHODS

We conducted an unblinded, single-center, EHR-nested, parallel-group, superiority randomized controlled trial comparing EHR-nested reminders with usual care in adult patients receiving lithium maintenance therapy for mood disorders. The primary outcome was the achievement of therapeutically appropriate serum lithium levels between 0.4 and 1.0 mEq/L at 18 months after enrollment. The key secondary outcomes are included as follows: the number of serum lithium level monitoring except for the first and final monitoring; exacerbation of the mood disorder during the study period, defined by hospitalization, increase in lithium dose, addition of antipsychotic drugs or mood stabilizers, or addition or increase of antidepressants; adherence defined by the proportion of days covered by lithium carbonate prescription during the study period.

RESULTS

A total of 111 patients were enrolled in this study. A total of 56 patients were assigned to the reminder group, and 55 patients were assigned to the usual care group. At the follow-up, 38 (69.1%) patients in the reminder group and 33 (60.0%) patients in the usual care group achieved the primary outcome (odds ratio 2.14, 95% CI 0.82-5.58, P=.12). The median number of serum lithium monitoring was 2 in the reminder group and 0 in the usual care group (rate ratio 3.62; 95% CI 2.47-5.29, P<.001). The exacerbation of mood disorders occurred in 17 (31.5%) patients in the reminder group and in 16 (34.8%) patients in the usual care group (odds ratio 0.97, 95% CI 0.42-2.28, P=.95).

CONCLUSIONS

We found insufficient evidence for an EHR-nested reminder to increase the achievement of therapeutic serum lithium concentrations. However, the number of monitoring increased with relatively simple and inexpensive intervention. The EHR-based reminders may be useful to improve quality of care for patients on lithium maintenance therapy, and they have potentials to be applied to other problems.

TRIAL REGISTRATION

University Hospital Medical Information Network Clinical Trials Registry UMIN000033633; https://tinyurl.com/5n7wtyav.

Usability and Clinician Acceptance of a Deep Learning-Based Clinical Decision Support Tool for Predicting Glaucomatous Visual Field Progression

PRCIS

We updated a clinical decision support tool integrating predicted visual field (VF) metrics from an artificial intelligence model and assessed clinician perceptions of the predicted VF metric in this usability study.

PURPOSE

To evaluate clinician perceptions of a prototyped clinical decision support (CDS) tool that integrates visual field (VF) metric predictions from artificial intelligence (AI) models.

METHODS

Ten ophthalmologists and optometrists from the University of California San Diego participated in 6 cases from 6 patients, consisting of 11 eyes, uploaded to a CDS tool ("GLANCE", designed to help clinicians "at a glance"). For each case, clinicians answered questions about management recommendations and attitudes towards GLANCE, particularly regarding the utility and trustworthiness of the AI-predicted VF metrics and willingness to decrease VF testing frequency.

MAIN OUTCOMES AND MEASURES

Mean counts of management recommendations and mean Likert scale scores were calculated to assess overall management trends and attitudes towards the CDS tool for each case. In addition, system usability scale scores were calculated.

RESULTS

The mean Likert scores for trust in and utility of the predicted VF metric and clinician willingness to decrease VF testing frequency were 3.27, 3.42, and 2.64, respectively (1=strongly disagree, 5=strongly agree). When stratified by glaucoma severity, all mean Likert scores decreased as severity increased. The system usability scale score across all responders was 66.1±16.0 (43rd percentile).

CONCLUSIONS

A CDS tool can be designed to present AI model outputs in a useful, trustworthy manner that clinicians are generally willing to integrate into their clinical decision-making. Future work is needed to understand how to best develop explainable and trustworthy CDS tools integrating AI before clinical deployment.

Physician-Peer Relationships and Patient Experiences With Specialist Care

IMPORTANCE

Peer relationships may motivate physicians to aspire to high professional standards but have not been a major focus of quality improvement efforts.

OBJECTIVE

To determine whether peer relationships between primary care physicians (PCPs) and specialists formed during training motivate improved specialist care for patients.

DESIGN, SETTING, AND PARTICIPANTS

In this quasi-experimental study, difference-in-differences analysis was used to estimate differences in experiences with specialist care reported by patients of the same PCP for specialists who did vs did not co-train with the PCP, controlling for any differences in patient ratings of the same specialists in the absence of co-training ties. Specialist visits resulting from PCP referrals from 2016 to 2019 in a large health system were analyzed, including a subset of undirected referrals in which PCPs did not specify a specialist. Data were collected from January 2016 to December 2019 and analyzed from March 2020 to October 2022.

EXPOSURE

The exposure was PCP-specialist overlap in training (medical school or postgraduate medical) at the same institution for at least 1 year (co-training).

MAIN OUTCOMES AND MEASURES

Composite patient experience rating of specialist care constructed from Press Ganey's Medical Practice Survey.

RESULTS

Of 9920 specialist visits for 8655 patients (62.9% female; mean age, 57.4 years) with 502 specialists in 13 specialties, 3.1% (306) involved PCP-specialist dyads with a co-training tie. Co-training ties between PCPs and specialists were associated with a 9.0 percentage point higher adjusted composite patient rating of specialist care (95% CI, 5.6-12.4 percentage points; P < .001), analogous to improvement from the median to the 91st percentile of specialist performance. This association was stronger for PCP-specialist dyads with full temporal overlap in training (same class or cohort) and consistently strong for 9 of 10 patient experience items, including clarity of communication and engagement in shared decision-making. In secondary analyses of objective markers of altered specialist practice in an expanded sample of visits not limited by the availability of patient experience data, co-training was associated with changes in medication prescribing, suggesting behavioral changes beyond interpersonal communication. Patient characteristics varied minimally by co-training status of PCP-specialist dyads. Results were similar in analyses restricted to undirected referrals (in which PCPs did not specify a specialist). Concordance between PCPs and specialists in physician age, sex, medical school graduation year, and training institution (without requiring temporal overlap) was not associated with better care experiences.

CONCLUSIONS AND RELEVANCE

In this quasi-experimental study, PCP-specialist co-training elicited changes in specialist care that substantially improved patient experiences, suggesting potential gains from strategies encouraging the formation of stronger physician-peer relationships.

Review of glaucoma medication adherence monitoring in the digital health era

Current glaucoma treatments aim to lower intraocular pressure, often with topical ocular hypotensive medications. Unfortunately, the effectiveness of these medications depends on sustained patient adherence to regimens which may involve instilling multiple medications several times daily. Patient adherence to glaucoma medications is often low. Recent innovations in digital sensor technologies have been leveraged to confirm eyedrop medication usage in real-time and relay this information back to providers. Some sensors have also been designed to deliver medication reminders and notifications as well as assist with correct eyedrop administration technique. Here, we review recent innovations targeted at improving glaucoma medication adherence and discuss their limitations.

Utilization of a Best Practice Alert (BPA) at Point-of-Care for Recruitment into a US-Based Autism Research Study

Provider referral is one of the most influential factors in research recruitment. To ease referral burden on providers, we adapted the Best Practice Alert (BPA) in the EPIC Electronic Health Record and assessed its utility in recruiting pediatric patients with autism spectrum disorder for the national SPARK study. During a year-long surveillance, 1203 (64.0%) patients were Interested in SPARK and 223 enrolled. Another 754 participants not recruited via the BPA also enrolled; 35.5% of these participants completed their participation compared to 58.3% of BPA-referred participants. Results suggest that (a) a BPA can successfully engage providers in the study-referral process and (b) families who learn about research through their providers may be more engaged and effectively retained.

Clinician Perspectives on Barriers and Enablers to Implementing an Inpatient Oncology Early Warning System: A Mixed-Methods Study

PURPOSE

To elicit end-user and stakeholder perceptions regarding design and implementation of an inpatient clinical deterioration early warning system (EWS) for oncology patients to better fit routine clinical practices and enhance clinical impact.

METHODS

In an explanatory-sequential mixed-methods study, we evaluated a stakeholder-informed oncology early warning system (OncEWS) using surveys and semistructured interviews. Stakeholders were physicians, advanced practice providers (APPs), and nurses. For qualitative data, we used grounded theory and thematic content analysis via the constant comparative method to identify determinants of OncEWS implementation.

RESULTS

Survey respondents generally agreed that an oncology-focused EWS could add value beyond clinical judgment, with nurses endorsing this notion significantly more strongly than other clinicians (nurse: median 5 on a 6-point scale [6 = strongly agree], interquartile range 4-5; doctors/advanced practice providers: 4 [4-5]; P = .005). However, some respondents would not trust an EWS to identify risk accurately (n = 36 [42%] somewhat or very concerned), while others were concerned that institutional culture would not embrace such an EWS (n = 17 [28%]).Interviews highlighted important aspects of the EWS and the local context that might facilitate implementation, including (1) a model tailored to the subtleties of oncology patients, (2) transparent model information, and (3) nursing-centric workflows. Interviewees raised the importance of sepsis as a common and high-risk deterioration syndrome.

CONCLUSION

Stakeholders prioritized maximizing the degree to which the OncEWS is understandable, informative, actionable, and workflow-complementary, and perceived these factors to be key for translation into clinical benefit.

Clinical trial recruitment in primary care: exploratory factor analysis of a questionnaire to measure barriers and facilitators to primary care providers' involvement

BACKGROUND

Recruitment of sufficient participants for clinical trials remains challenging. Primary care is an important avenue for patient recruitment but is underutilized. We developed and pilot tested a questionnaire to measure relevant barriers and facilitators to primary care providers' involvement in recruiting patients for clinical trials.

METHODS

Prior research informed the development of the questionnaire. The initial instrument was revised using feedback obtained from cognitive interviews. We invited all primary care providers practicing within the University of Utah Health system to complete the revised questionnaire. We used a mixed-mode design to collect paper responses via in-person recruitment and email contacts to collect responses online. Descriptive statistics, exploratory factor analysis, Cronbach's alpha, and multivariable regression analyses were conducted.

RESULTS

Sixty-seven primary care providers participated in the survey. Exploratory factor analysis suggested retaining five factors, representing the importance of clinical trial recruitment in providers' professional identity, clinic-level interventions to facilitate referral, patient-related barriers, concerns about patient health management, and knowledge gaps. The five factors exhibited good or high internal consistency reliability. Professional identity and clinic-level intervention factors were significant predictors of providers' intention to participate in clinical trial recruitment activities.

CONCLUSIONS

Results of this exploratory analysis provide preliminary evidence of the internal structure, internal consistency reliability, and predictive validity of the questionnaire to measure factors relevant to primary care providers' involvement in clinical trial recruitment.

Diachronic and synchronic variation in the performance of adaptive machine learning systems: the ethical challenges

OBJECTIVES

Machine learning (ML) has the potential to facilitate "continual learning" in medicine, in which an ML system continues to evolve in response to exposure to new data over time, even after being deployed in a clinical setting. In this article, we provide a tutorial on the range of ethical issues raised by the use of such "adaptive" ML systems in medicine that have, thus far, been neglected in the literature.

TARGET AUDIENCE

The target audiences for this tutorial are the developers of ML AI systems, healthcare regulators, the broader medical informatics community, and practicing clinicians.

SCOPE

Discussions of adaptive ML systems to date have overlooked the distinction between 2 sorts of variance that such systems may exhibit-diachronic evolution (change over time) and synchronic variation (difference between cotemporaneous instantiations of the algorithm at different sites)-and underestimated the significance of the latter. We highlight the challenges that diachronic evolution and synchronic variation present for the quality of patient care, informed consent, and equity, and discuss the complex ethical trade-offs involved in the design of such systems.

Assessing the use of a clinical decision support tool for pain management in primary care

Objective

Given time constraints, poorly organized information, and complex patients, primary care providers (PCPs) can benefit from clinical decision support (CDS) tools that aggregate and synthesize problem-specific patient information. First, this article describes the design and functionality of a CDS tool for chronic noncancer pain in primary care. Second, we report on the retrospective analysis of real-world usage of the tool in the context of a pragmatic trial.

Materials and methods

The tool known as OneSheet was developed using user-centered principles and built in the Epic electronic health record (EHR) of 2 health systems. For each relevant patient, OneSheet presents pertinent information in a single EHR view to assist PCPs in completing guideline-recommended opioid risk mitigation tasks, review previous and current patient treatments, view patient-reported pain, physical function, and pain-related goals.

Results

Overall, 69 PCPs accessed OneSheet 2411 times (since November 2020). PCP use of OneSheet varied significantly by provider and was highly skewed (site 1: median accesses per provider: 17 [interquartile range (IQR) 9-32]; site 2: median: 8 [IQR 5-16]). Seven "power users" accounted for 70% of the overall access instances across both sites. OneSheet has been accessed an average of 20 times weekly between the 2 sites.

Discussion

Modest OneSheet use was observed relative to the number of eligible patients seen with chronic pain.

Conclusions

Organizations implementing CDS tools are likely to see considerable provider-level variation in usage, suggesting that CDS tools may vary in their utility across PCPs, even for the same condition, because of differences in provider and care team workflows.

Using the Electronic Health Record User Context in Clinical Decision Support Criteria

BACKGROUND

Computerized clinical decision support (CDS) used in electronic health record systems (EHRs) has led to positive outcomes as well as unintended consequences, such as alert fatigue. Characteristics of the EHR session can be used to restrict CDS tools and increase their relevance, but implications of this approach are not rigorously studied.

OBJECTIVES

To assess the utility of using "login location" of EHR users-that is, the location they chose on the login screen-as a variable in the CDS logic.

METHODS

We measured concordance between user's login location and the location of the patients they placed orders for and conducted stratified analyses by user groups. We also estimated how often login location data may be stale or inaccurate.

RESULTS

One in five CDS alerts incorporated the EHR users' login location into their logic. Analysis of nearly 2 million orders placed by nearly 8,000 users showed that concordance between login location and patient location was high for nurses, nurse practitioners, and physician assistance (all >95%), but lower for fellows (77%) and residents (55%). When providers switched between patients in the EHR, they usually did not update their login location accordingly.

CONCLUSION

CDS alerts commonly incorporate user's login location into their logic. User's login location is often the same as the location of the patient the user is providing care for, but substantial discordance can be observed for certain user groups. While this may provide additional information that could be useful to the CDS logic, a substantial amount of discordance happened in specific user groups or when users appeared not to change their login location across different sessions. Those who design CDS alerts should consider a data-driven approach to evaluate the appropriateness of login location for each use case.

Aiding the prescriber: developing a machine learning approach to personalized risk modeling for chronic opioid therapy amongst US Army soldiers

The opioid epidemic is a major policy concern. The widespread availability of opioids, which is fueled by physician prescribing patterns, medication diversion, and the interaction with potential illicit opioid use, has been implicated as proximal cause for subsequent opioid dependence and mortality. Risk indicators related to chronic opioid therapy (COT) at the point of care may influence physicians' prescribing decisions, potentially reducing rates of dependency and abuse. In this paper, we investigate the performance of machine learning algorithms for predicting the risk of COT. Using data on over 12 million observations of active duty US Army soldiers, we apply machine learning models to predict the risk of COT in the initial months of prescription. We use the area under the curve (AUC) as an overall measure of model performance, and we focus on the positive predictive value (PPV), which reflects the models' ability to accurately target military members for intervention. Of the many models tested, AUC ranges between 0.83 and 0.87. When we focus on the top 1% of members at highest risk, we observe a PPV value of 8.4% and 20.3% for months 1 and 3, respectively. We further investigate the performance of sparse models that can be implemented in sparse data environments. We find that when the goal is to identify patients at the highest risk of chronic use, these sparse linear models achieve a performance similar to models trained on hundreds of variables. Our predictive models exhibit high accuracy and can alert prescribers to the risk of COT for the highest risk patients. Optimized sparse models identify a parsimonious set of factors to predict COT: initial supply of opioids, the supply of opioids in the month being studied, and the number of prescriptions for psychotropic medications. Future research should investigate the possible effects of these tools on prescriber behavior (e.g., the benefit of clinician nudging at the point of care in outpatient settings).

Upcoming and urgent challenges in critical care research based on COVID-19 pandemic experience

While the coronavirus disease 2019 (COVID-19) pandemic placed a heavy burden on healthcare systems worldwide, it also induced urgent mobilisation of research teams to develop treatments preventing or curing the disease and its consequences. It has, therefore, challenged critical care research to rapidly focus on specific fields while forcing critical care physicians to make difficult ethical decisions. This narrative review aims to summarise critical care research -from organisation to research fields- in this pandemic setting and to highlight opportunities to improve research efficiency in the future, based on what is learned from COVID-19. This pressure on research revealed, i.e., i/ the need to harmonise regulatory processes between countries, allowing simplified organisation of international research networks to improve their efficiency in answering large-scale questions; ii/ the importance of developing translational research from which therapeutic innovations can emerge; iii/ the need for improved triage and predictive scores to rationalise admission to the intensive care unit. In this context, key areas for future critical care research and better pandemic preparedness are artificial intelligence applied to healthcare, characterisation of long-term symptoms, and ethical considerations. Such collaborative research efforts should involve groups from both high and low-to-middle income countries to propose worldwide solutions. As a conclusion, stress tests on healthcare organisations should be viewed as opportunities to design new research frameworks and strategies. Worldwide availability of research networks ready to operate is essential to be prepared for next pandemics. Importantly, researchers and physicians should prioritise realistic and ethical goals for both clinical care and research.

Overcoming Decisional Gaps in High-Risk Prescribing by Junior Physicians Using Simulation-Based Training: Protocol for a Randomized Controlled Trial

BACKGROUND

Gaps between rational thought and actual decisions are increasingly recognized as a reason why people make suboptimal choices in states of heightened emotion, such as stress. These observations may help explain why high-risk medications continue to be prescribed to acutely ill hospitalized older adults despite widely accepted recommendations against these practices. Role playing and other efforts, such as simulation training, have demonstrated benefits to help people avoid decisional gaps but have not been tested to reduce overprescribing of high-risk medications.

OBJECTIVE

This study aims to evaluate the impact of a simulation-based training program designed to address decisional gaps on prescribing of high-risk medications compared with control.

METHODS

In this 2-arm pragmatic trial, we are randomizing at least 36 first-year medical resident physicians (ie, interns) who provide care on inpatient general medicine services at a large academic medical center to either intervention (simulation-based training) or control (online educational training). The intervention comprises a 40-minute immersive individual simulation training consisting of a reality-based patient care scenario in a simulated environment at the beginning of their inpatient service rotation. The simulation focuses on 3 types of high-risk medications, including benzodiazepines, antipsychotics, and sedative hypnotics (Z-drugs), in older adults, and is specifically designed to help the physicians identify their reactions and prescribing decisions in stressful situations that are common in the inpatient setting. The simulation scenario is followed by a semistructured debriefing with an expert facilitator. The trial's primary outcome is the number of medication doses for any of the high-risk medications prescribed by the interns to patients aged 65 years or older who were not taking one of the medications upon admission. Secondary outcomes include prescribing by all providers on the care team, being discharged on 1 of the medications, and prescribing of related medications (eg, melatonin, trazodone), or the medications of interest for the control intervention. These outcomes will be measured using electronic health record data.

RESULTS

Recruitment of interns began on March 29, 2021. Recruitment for the trial ended in Q42021, with follow-up completed by Q12022.

CONCLUSIONS

This trial will evaluate the impact of a simulation-based training program designed using behavioral science principles on prescribing of high-risk medications by junior physicians. If the intervention is shown to be effective, this approach could potentially be reproducible by others and for a broader set of behaviors.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04668248; https://clinicaltrials.gov/ct2/show/NCT04668248.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/31464.

Nursing implications of an early warning system implemented to reduce adverse events: a qualitative study

BACKGROUND

Unrecognised changes in a hospitalised patient's clinical course may lead to a preventable adverse event. Early warning systems (EWS) use patient data, such as vital signs, nursing assessments and laboratory values, to aid in the detection of early clinical deterioration. In 2018, an EWS programme was deployed at an academic hospital that consisted of a commercially available EWS algorithm and a centralised virtual nurse team to monitor alerts. Our objective was to understand the nursing perspective on the use of an EWS programme with centralised monitoring.

METHODS

We conducted and audio-recorded semistructured focus groups during nurse staff meetings on six inpatient units, stratified by alert frequency (high: >100 alerts/month; medium: 50-100 alerts/month; low: <50 alerts/month). Discussion topics included EWS programme experiences, perception of EWS programme utility and EWS programme implementation. Investigators analysed the focus group transcripts using a grounded theory approach.

RESULTS

We conducted 28 focus groups with 227 bedside nurses across all shifts. We identified six principal themes: (1) Alert timeliness, nurses reported being aware of the patient's deterioration before the EWS alert, (2) Lack of accuracy, nurses perceived most alerts as false positives, (3) Workflow interruptions caused by EWS alerts, (4) Questions of actionability of alerts, nurses were often uncertain about next steps, (5) Concerns around an underappreciation of core nursing skills via reliance on the EWS programme and (6) The opportunity cost of deploying the EWS programme.

CONCLUSION

This qualitative study of nurses demonstrates the importance of earning user trust, ensuring timeliness and outlining actionable next steps when implementing an EWS. Careful attention to user workflow is required to maximise EWS impact on improving hospital quality and patient safety.

Evaluation of an Electronic Health Record Alert to Improve Screening and Management of Cardiovascular Disease and Stroke Factors in a High-Risk Population

OBJECTIVES

Cardiovascular disease and stroke risk factor screening and management by primary care providers (PCPs) have a significant impact on their patients' health. The objective of this study was to investigate the effectiveness of an electronic health record (EHR) cardiovascular disease and stroke risk alert in improving the ability of PCPs to manage risk factors among women and men aged 45 years and older.

METHODS

PCPs at a tertiary care hospital were randomized. The intervention group received an EHR alert, which calculated the individual patient risk and provided an order set incorporating the American Heart Association and American Stroke Association guidelines, whereas the control group used the EHR in the usual manner. Multilevel analysis compared the rate of prescriptions between the intervention and control groups.

RESULTS

A total of 23 PCPs were randomized: 12 in the intervention group and 11 in the control group, attending to 7190 patients between September 2016 and January 2017. None of the providers in the intervention group used the programmed order set. Intervention group providers were significantly more likely to prescribe smoking cessation medication to women than to the control group (adjusted odds ratio 2.37, 95% confidence interval 1.23-4.57). There were no statistically significant differences between the intervention and control groups in the rate of other medication prescriptions.

CONCLUSIONS

As measured by prescriptions for medications, other than those for smoking cessation, the EHR alert was not shown to be successful in increasing the management of high-risk patients. Physicians receiving numerous messages in the EHR may experience alert desensitization.

Habit and Automaticity in Medical Alert Override: Cohort Study

Background

Prior literature suggests that alert dismissal could be linked to physicians’ habits and automaticity. The evidence for this perspective has been mainly observational data. This study uses log data from an electronic medical records system to empirically validate this perspective.

Objective

We seek to quantify the association between habit and alert dismissal in physicians.

Methods

We conducted a retrospective analysis using the log data comprising 66,049 alerts generated from hospitalized patients in a hospital from March 2017 to December 2018. We analyzed 1152 physicians exposed to a specific clinical support alert triggered in a hospital’s electronic medical record system to estimate the extent to which the physicians’ habit strength, which had been developed from habitual learning, impacted their propensity toward alert dismissal. We further examined the association between a physician’s habit strength and their subsequent incidences of alert dismissal. Additionally, we recorded the time taken by the physician to respond to the alert and collected data on other clinical and environmental factors related to the alerts as covariates for the analysis.

Results

We found that a physician’s prior dismissal of alerts leads to their increased habit strength to dismiss alerts. Furthermore, a physician’s habit strength to dismiss alerts was found to be positively associated with incidences of subsequent alert dismissals after their initial alert dismissal. Alert dismissal due to habitual learning was also found to be pervasive across all physician ranks, from junior interns to senior attending specialists. Further, the dismissal of alerts had been observed to typically occur after a very short processing time. Our study found that 72.5% of alerts were dismissed in under 3 seconds after the alert appeared, and 13.2% of all alerts were dismissed in under 1 second after the alert appeared. We found empirical support that habitual dismissal is one of the key factors associated with alert dismissal. We also found that habitual dismissal of alerts is self-reinforcing, which suggests significant challenges in disrupting or changing alert dismissal habits once they are formed.

Conclusions

Habitual tendencies are associated with the dismissal of alerts. This relationship is pervasive across all levels of physician rank and experience, and the effect is self-reinforcing.

Clinical Decision Support Systems to Reduce Unnecessary Clostridoides difficile Testing Across Multiple Hospitals

BACKGROUND

Inappropriate C. difficile testing has adverse consequences for the patient, hospital, and public health. Computerized Clinical Decision Supports (CCDS) in the Electronic Health Record (EHR) may reduce C. difficile test ordering; however, effectiveness of different approaches, ease of use, and best fit into the healthcare providers' (HCP) workflow, are not well understood.

METHODS

Nine academic and 6 community U.S. hospitals participated in this 2-year cohort study. CCDS (hard- or soft-stop) triggered when duplicate C. difficile test order attempted, or if laxatives were recently received. The primary outcome was the difference in testing rates pre- and post-CCDS interventions, using incident rate ratios (IRR) and mixed effect Poisson regression models. We performed qualitative evaluation (contextual inquiry, interviews, focus groups) based on a human factors model. We identified themes using a codebook with primary- and sub-nodes.

RESULTS

In 9 hospitals implementing hard-stop CCDS and 4 hospitals implementing soft-stop CCDS, C. difficile testing IRR reduction was 33% (95% CI, 30-36%), and 23% (95% CI 21-25%), respectively. Two hospitals implemented a non-EMR based human intervention with IRR reduction of 21% (95% CI 15-28%). HCPs reported generally favorable experiences, and highlighted time efficiencies such as inclusion of the patients most recent laxative administration on the CCDS. Organizational factors including hierarchical cultures, and communication between HCPs caring for the same patient, impact CCDS acceptance and integration.

CONCLUSIONS

CCDS reduced unnecessary C. difficile testing and were perceived positively by HCPs when integrated into their workflow, and when displayed relevant patient specific information needed for decision-making.

Improving annual albuminuria testing for individuals with diabetes

Background

Annual albuminuria screening detects the early stages of nephropathy in individuals with diabetes. Because early detection of albuminuria allows for interventions that lower the risk of developing chronic kidney disease, guidelines recommend annual testing for all individuals with type 2 diabetes mellitus and for those with type 1 diabetes for at least 5 years. However, at the Eskind Diabetes Clinic at the Vanderbilt University Medical Center, testing occurred less frequently than desired.

Methods

A quality improvement team first analysed the clinic’s processes, identifying the lack of a systematic approach to testing as the likely cause for the low rate. The team then implemented two successive interventions in a pilot of patients seen by nurse practitioners in the clinic. In the first intervention, staff used a dashboard within the electronic health record while triaging each patient, pending an albuminuria order if testing had not been done within the past year. In the second intervention, clinic leadership sent daily reminders to the triage staff. A statistical process control chart tracked monthly testing rates.

Results

After 6 months, annual albuminuria testing increased from a baseline of 69% to 82%, with multiple special-cause signals in the control chart.

Conclusions

This project demonstrates that a series of simple interventions can significantly impact annual albuminuria testing. This project’s success likely hinged on using an existing workflow to systematically determine if a patient was due for testing and prompting the provider to sign a pended order for an albuminuria test. Other diabetes/endocrinology and primary care clinics can likely implement a similar process and so improve testing rates in other settings. When coupled with appropriate interventions to reduce the development of chronic kidney disease, such interventions would improve patient outcomes, in addition to better adhering to an established quality metric.

Selecting EHR-driven recruitment strategies: An evidence-based decision guide

Participant recruitment for research is a persistent bottleneck that can be improved by leveraging electronic health records (EHRs). Despite emerging evidence for various EHR-driven approaches, guidance for those attempting to select and use such approaches is limited. The national Recruitment Innovation Center established the EHR Recruitment Consult Resource (ERCR) service line to support multisite studies through implementation of EHR-driven recruitment strategies. As the ERCR, we evolved a guide through 17 consultations over 3 years with multisite studies recruiting in diverse biomedical research domains. We assessed literature and engaged domain experts to identify five key EHR-driven recruitment strategies: direct to patient messages, candidate lists for mailings/calls, direct to research alerts, point of care alerts, and participant registries. Differentiating factors were grouped into factors of study population, study protocol and recruitment workflows, and recruitment site capabilities. The decision matrix indicates acceptable or preferred strategies based on the differentiating factors. Across the ERCR consultations, candidate lists for mailing or calls were most common, participant registries were least frequently recommended, and for some studies no EHR-driven recruitment was recommended. Comparative effectiveness research is needed to refine further evidence for these and potentially new strategies to come.

Improving antimicrobial prescribing for upper respiratory infections in the emergency department: Implementation of peer comparison with behavioral feedback

OBJECTIVE

To reduce inappropriate antibiotic prescribing for acute respiratory infections (ARIs) by employing peer comparison with behavioral feedback in the emergency department (ED).

DESIGN

A controlled before-and-after study.

SETTING

The study was conducted in 5 adult EDs at teaching and community hospitals in a health system.

PATIENTS

Adults presenting to the ED with a respiratory condition diagnosis code. Hospitalized patients and those with a diagnosis code for a non-respiratory condition for which antibiotics are or may be warranted were excluded.

INTERVENTIONS

After a baseline period from January 2016 to March 2018, 3 EDs implemented a feedback intervention with peer comparison between April 2018 and December 2019 for attending physicians. Also, 2 EDs in the health system served as controls. Using interrupted time series analysis, the inappropriate ARI prescribing rate was calculated as the proportion of antibiotic-inappropriate ARI encounters with a prescription. Prescribing rates were also evaluated for all ARIs. Attending physicians at intervention sites received biannual e-mails with their inappropriate prescribing rate and had access to a dashboard that was updated daily showing their performance relative to their peers.

RESULTS

Among 28,544 ARI encounters, the inappropriate prescribing rate remained stable at the control EDs between the 2 periods (23.0% and 23.8%). At the intervention sites, the inappropriate prescribing rate decreased significantly from 22.0% to 15.2%. Between periods, the overall ARI prescribing rate was 38.1% and 40.6% in the control group and 35.9% and 30.6% in the intervention group.

CONCLUSIONS

Behavioral feedback with peer comparison can be implemented effectively in the ED to reduce inappropriate prescribing for ARIs.

Improving vitamin D testing and supplementation in children with newly diagnosed cancer: A quality improvement initiative at Rady Children's Hospital San Diego

BACKGROUND

Vitamin D deficiency and insufficiency have been associated with poorer health outcomes. Children with cancer are at high risk for vitamin D deficiency and insufficiency. At our institution, we identified high variability in vitamin D testing and supplementation in this population. Of those tested, 65% were vitamin D deficient/insufficient. We conducted a quality improvement (QI) initiative with aim to improve vitamin D testing and supplementation among children aged 2-18 years with newly diagnosed cancer to ≥80% over 6 months.

METHODS

An inter-professional team reviewed baseline data, then developed and implemented interventions using Plan-Do-Study-Act (PDSA) cycles. Barriers were identified using QI tools, including lack of automated triggers for testing and inconsistent supplementation criteria and follow-up testing post supplementation. Interventions included an institutional vitamin D guideline, clinical decision-making tree for vitamin D deficiency, insufficiency and sufficiency, electronic medical record triggers, and automated testing options.

RESULTS

Baseline: N = 26 patients, four (15%) had baseline vitamin D testing; two (8%) received appropriate supplementation. Postintervention: N = 33 patients; 32 (97%) had baseline vitamin D testing; 33 (100%) received appropriate supplementation and completed follow-up testing timely (6-8 weeks post supplementation). Change was sustained over 24 months.

CONCLUSIONS

We achieved and sustained our aim for vitamin D testing and supplementation in children with newly diagnosed cancer through inter-professional collaboration of hematology/oncology, endocrinology, hospital medicine, pharmacy, nursing, and information technology. Future PDSA cycles will address patient compliance with vitamin D supplementation and impact on patients' vitamin D levels.

The Impact of Outpatient Laboratory Alerting Mechanisms in Patients with AKI

Background

AKI is an abrupt decrease in kidney function associated with significant morbidity and mortality. Electronic notifications of AKI have been utilized in patients who are hospitalized, but their efficacy in the outpatient setting is unclear.

Methods

We evaluated the effect of two outpatient interventions: an automated comment on increasing creatinine results (intervention I; 6 months; n=159) along with an email to the provider (intervention II; 3 months; n=105), compared with a control (baseline; 6 months; n=176). A comment was generated if a patient's creatinine increased by >0.5 mg/dl (previous creatinine ≤2.0 mg/dl) or by 50% (previous creatinine >2.0 mg/dl) within 180 days. Process measures included documentation of AKI and clinical actions. Clinical outcomes were defined as recovery from AKI within 7 days, prolonged AKI from 8 to 89 days, and progression to CKD with in 120 days.

Results

Providers were more likely to document AKI in interventions I (P=0.004; OR, 2.80; 95% CI, 1.38 to 5.67) and II (P=0.01; OR, 2.66; 95% CI, 1.21 to 5.81). Providers were also more likely to discontinue nephrotoxins in intervention II (P<0.001; OR, 4.88; 95% CI, 2.27 to 10.50). The median time to follow-up creatinine trended shorter among patients with AKI documented (21 versus 42 days; P=0.11). There were no significant differences in clinical outcomes.

Conclusions

An automated comment was associated with improved documented recognition of AKI and the additive intervention of an email alert was associated with increased discontinuation of nephrotoxins, but neither improved clinical outcomes. Translation of these findings into improved outcomes may require corresponding standardization of clinical practice protocols for managing AKI.

The appropriateness of clinical decision support systems alerts in supporting clinical workflows: A systematic review

A CDSS generates a high number of inappropriate alerts that interrupt the clinical workflow. As a result, clinicians silence, disable, or ignore alerts, thereby undermining patient safety. Therefore, the effectiveness and appropriateness of CDSS alerts need to be evaluated. A systematic review was carried out to identify the factors that affect CDSS alert appropriateness in supporting clinical workflow. Seven electronic databases (PubMed, Scopus, ACM, Science Direct, IEEE, Ovid Medline, and Ebscohost) were searched for English language articles published between 1997 and 2018. Seventy six papers met the inclusion criteria, of which 26, 24, 15, and 11 papers are retrospective cohort, qualitative, quantitative, and mixed-method studies, respectively. The review highlights various factors influencing the appropriateness and efficiencies of CDSS alerts. These factors are categorized into technology, human, organization, and process aspects using a combination of approaches, including socio-technical framework, five rights of CDSS, and Lean. Most CDSS alerts were not properly designed based on human factor methods and principles, explaining high alert overrides in clinical practices. The identified factors and recommendations from the review may offer valuable insights into how CDSS alerts can be designed appropriately to support clinical workflow.

Machine Learning to Predict Tamoxifen Nonadherence Among US Commercially Insured Patients With Metastatic Breast Cancer

PURPOSE

Adherence to tamoxifen citrate among women diagnosed with metastatic breast cancer can improve survival and minimize recurrence. This study aimed to use real-world data and machine learning (ML) methods to classify tamoxifen nonadherence.

METHODS

A cohort of women diagnosed with metastatic breast cancer from 2012 to 2017 were identified from IBM MarketScan Commercial Claims and Encounters and Medicare claims databases. Patients with < 80% proportion of days coverage in the year following treatment initiation were classified as nonadherent. Training and internal validation cohorts were randomly generated (4:1 ratio). Clinical procedures, comorbidity, treatment, and health care encounter features in the year before tamoxifen initiation were used to train logistic regression, boosted logistic regression, random forest, and feedforward neural network models and were internally validated on the basis of area under receiver operating characteristic curve. The most predictive ML approach was evaluated to assess feature importance.

RESULTS

A total of 3,022 patients were included with 40% classified as nonadherent. All models had moderate predictive accuracy. Logistic regression (area under receiver operating characteristic 0.64) was interpreted with 94% sensitivity (95% CI, 89 to 92) and 0.31 specificity (95% CI, 29 to 33). The model accurately classified adherence (negative predictive value 89%) but was nondiscriminate for nonadherence (positive predictive value 48%). Variable importance identified top predictive factors, including age ≥ 55 years and pretreatment procedures (lymphatic nuclear medicine, radiation oncology, and arterial surgery).

CONCLUSION

ML using baseline administrative data predicts tamoxifen nonadherence. Screening at treatment initiation may support personalized care, improve health outcomes, and minimize cost. Baseline claims may not be sufficient to discriminate adherence. Further validation with enriched longitudinal data may improve model performance.

Physician Well-being and the Future of Health Information Technology

The issue of clinician burnout has become a growing concern in health care, with an increased emphasis on health information technology as a contributing factor. Technology-mediated stresses have arisen with the electronic health record, and we can anticipate new and different impacts from future information tools. This article discusses technology's pivotal role in physician well-being, not only in the quality of its design but also through its capacity to enable future models of care that are more manageable for physicians and more effective for patients. Three general aims along with specific efforts are proposed to benefit physician well-being in technology-mediated work.

Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review

OBJECTIVE

Real-world data (RWD), defined as routinely collected healthcare data, can be a potential catalyst for addressing challenges faced in clinical trials. We performed a scoping review of database-specific RWD applications within clinical trial contexts, synthesizing prominent uses and themes.

MATERIALS AND METHODS

Querying 3 biomedical literature databases, research articles using electronic health records, administrative claims databases, or clinical registries either within a clinical trial or in tandem with methodology related to clinical trials were included. Articles were required to use at least 1 US RWD source. All abstract screening, full-text screening, and data extraction was performed by 1 reviewer. Two reviewers independently verified all decisions.

RESULTS

Of 2020 screened articles, 89 qualified: 59 articles used electronic health records, 29 used administrative claims, and 26 used registries. Our synthesis was driven by the general life cycle of a clinical trial, culminating into 3 major themes: trial process tasks (51 articles); dissemination strategies (6); and generalizability assessments (34). Despite a diverse set of diseases studied, <10% of trials using RWD for trial process tasks evaluated medications or procedures (5/51). All articles highlighted data-related challenges, such as missing values.

DISCUSSION

Database-specific RWD have been occasionally leveraged for various clinical trial tasks. We observed underuse of RWD within conducted medication or procedure trials, though it is subject to the confounder of implicit report of RWD use.

CONCLUSION

Enhanced incorporation of RWD should be further explored for medication or procedure trials, including better understanding of how to handle related data quality issues to facilitate RWD use.

Improving timeliness of hepatitis B vaccine administration in an urban safety net level III NICU

OBJECTIVE

To avoid preventable consequences of perinatal hepatitis B infection, all infants should be given hepatitis B vaccine (HBV) within 24 hours of birth if birth weight is ≥2 kg and at 30 days of life or at discharge if <2 kg, to provide highest seroprotection rates while ensuring universal vaccination prior to discharge. We aimed to achieve timely HBV administration in >80% of eligible infants in both birthweight groups and decrease infants discharged home without receiving HBV to <1% over an 18-month period and sustain results for an additional 15 months.

METHODS

Data were collected from June 2016 to May 2020 in a level III neonatal intensive care unit. A multidisciplinary team identified barriers and interventions through Plan-Do-Study-Act cycles from September 2017 to February 2019: using pharmacists as champions, overcoming legal barriers, staff education and best practice alerts (BPAs) embedded in electronic health records. Statistical process control (SPC) p charts were used to evaluate the primary outcome measure, monthly percentage of infants receiving timely HBV administration stratified by birthweight categories (≥2 and <2 kg). For infants receiving HBV outside the time frame, absolute difference of timeliness was calculated.

RESULTS

Mean timely HBV administration improved from 45% to 95% (≥2 kg) and from 45% to 85% (<2 kg) with special cause variation in SPC charts. Infants discharged without receiving HBV decreased from 4.6% to 0.22%. Of those given HBV outside the recommended time frame, median absolute time between recommended and actual administration time decreased significantly: from 3.5 days (IQR 1.6, 8.6) to 0.3 day (IQR 0.1, 0.8) (p<0.001) in ≥2 kg group and from 6 days (IQR 1, 15) to 1 day (IQR 1, 6.5) (p=0.009) in <2 kg group.

CONCLUSIONS

Using a multidisciplinary approach, we significantly improved and sustained timely HBV administration and nearly eliminated infants discharged home without receiving HBV. Pharmacists as champions and BPAs were critical to our success.

Nurse Cognition, Decision Support, and Barcode Medication Administration: A Conceptual Framework for Research, Practice, and Education

This article synthesizes theoretical perspectives related to nurse cognition. We present a conceptual model that can be used by multiple stakeholders to study and contemplate how nurses use clinical decision support systems, and specifically, Barcode-Assisted Medication Administration, to make decisions during the delivery of care. Theoretical perspectives integrated into the model include dual process theory, the Cognitive Continuum Theory, human factors engineering, and the Recognition-Primed Decision model. The resulting framework illustrates the process of nurse cognition during Barcode-Assisted Medication Administration. Additionally, the model includes individual or human and environmental factors that may influence nurse cognition and decision making. It is important to consider the influence of individual, human, and environmental factors on the process of nurse cognition and decision making. Specifically, it is necessary to explore the impact of heuristics and biases on clinician decision making, particularly related to the development of alarm and alert fatigue. Aided by the proposed framework, stakeholders may begin to identify heuristics and cognitive biases that influence the decision of clinicians to accept or override a clinical decision support system alert and whether heuristics and biases are associated with inappropriate alert override.

Evaluation of Clinical Decision Support to Reduce Sedative-Hypnotic Prescribing in Older Adults

OBJECTIVE

We sought to characterize the performance of inpatient and outpatient computerized clinical decision support (CDS) alerts aimed at reducing inappropriate benzodiazepine and nonbenzodiazepine sedative medication prescribing in older adults 18 months after implementation.

METHODS

We reviewed the performance of two CDS alerts in the outpatient and inpatient settings in 2019. To examine the alerts' effectiveness, we analyzed metrics including overall alert adherence, provider-level adherence, and reasons for alert trigger and override.

RESULTS

In 2019, we identified a total of 14,534 and 4,834 alerts triggered in the outpatient and inpatient settings, respectively. Providers followed only 1% of outpatient and 3% of inpatient alerts. Most alerts were ignored (68% outpatient and 60% inpatient), while providers selected to override the remaining alerts. In each setting, the top 2% of clinicians were responsible for approximately 25% of all ignored or overridden alerts. However, a small proportion of clinicians (2% outpatient and 4% inpatient) followed the alert at least half of the time and accounted for a disproportionally large fraction of the total followed alerts. Our analysis of the free-text comments revealed that many alerts were to continue outpatient prescriptions or for situational anxiety.

CONCLUSION

Our findings highlight the importance of evaluation of CDS performance after implementation. We found large variation in response to the inpatient and outpatient alerts, both with respect to follow and ignore rates. Reevaluating the alert design by providing decision support by indication may be more helpful and may reduce alert fatigue.

Machine Learning and Clinical Informatics for Improving HIV Care Continuum Outcomes

PURPOSE OF REVIEW

This manuscript reviews the use of electronic medical record (EMR) data for HIV care and research along the HIV care continuum with a specific focus on machine learning methods and clinical informatics interventions.

RECENT FINDINGS

EMR-based clinical decision support tools and electronic alerts have been effectively utilized to improve HIV care continuum outcomes. Accurate EMR-based machine learning models have been developed to predict HIV diagnosis, retention in care, and viral suppression. Natural language processing (NLP) of clinical notes and data sharing between healthcare systems and public health agencies can enhance models for identifying people living with HIV who are undiagnosed or in need of relinkage to care. Challenges related to using these technologies include inconsistent EMR documentation, alert fatigue, and the potential for bias. Clinical informatics and machine learning models are promising tools for improving HIV care continuum outcomes. Future research should focus on methods for combining EMR data with additional data sources (e.g., social media, geospatial data) and studying how to effectively implement predictive models for HIV care into clinical practice.