Alert fatigue in electronic health records

Reduction in Unindicated Cervical Cancer Screening in Adolescents in a Large Health Care System

OBJECTIVES/PURPOSE

Evidence-based guidelines recommend against screening for cervical cancer (Pap testing) in average-risk adolescents <21 years old. Despite this, many still undergo unindicated screenings with subsequent detrimental reproductive health and economic consequences. Our aim was to reduce unindicated cervical cancer screening in individuals <21 years old in a large health care system by utilizing an electronic provider notification.

METHODS

Starting in July 2020, a Best Practice Advisory (BPA) appeared in the electronic medical record (EMR) if providers ordered Pap testing on individuals <21 years old. This BPA reiterated that screening was not indicated for average-risk adolescents and prompted users to choose an indication if they wanted to proceed.A retrospective chart review, pre/post intervention study was performed comparing individuals <21 years old with Pap testing performed before and after intervention (January 2019-June 2020 and July 2020-June 2021, respectively). Patient characteristics were extracted from the EMR and analyzed using Fisher exact tests, Kruskal-Wallis tests, and logistic regression.

RESULTS

There were 140 subjects included: 106 preintervention and 34 postintervention. There were no differences in baseline characteristics. Neither Pap nor human papillomavirus testing results differed between the groups. Preintervention, 6.6% of cytology tests were indicated compared to 20.6% postintervention ( p = .042). The proportion of indicated human papillomavirus testing did not differ preintervention and postintervention at 65% and 45%, respectively ( p = .295). The overall reduction in unindicated cervical cancer screening postintervention was 13.9% (95% CI = 4.0-23.7).

CONCLUSIONS

We demonstrated that incorporating a BPA to the EMR reduces unindicated cervical cancer screening.

Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool

OBJECTIVES

 To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high 1-year mortality risk.

METHODS

 We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low risk of 1-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality.

RESULTS

 Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (standard deviation [SD]) age of 69.3 (17.5) years, and a mean (SD) Charlson Comorbidity Index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high risk (68.7% concordance). For 105 patients whom PCPs deemed SIC unnecessary, the tool classified 83 as low risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640).

CONCLUSIONS

 A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

Implementation of a myasthenia gravis drug-disease interaction clinical decision support tool reduces prescribing of high-risk medications

INTRODUCTION/AIMS

High-risk medication exposure is a modifiable risk factor for myasthenic exacerbation and crisis. We evaluated whether real-time electronic clinical decision support (CDS) was effective in reducing the rate of prescribing potentially high-risk medications to avoid or use with caution in patients with myasthenia gravis.

METHODS

An expert panel reviewed the available drug-disease pairings and associated severity levels to activate the alerts for CDS. All unique alerts activated in both inpatient and outpatient contexts were analyzed over a two-year period. Clinical context, alert severity, medication class, and alert action were collected. The primary outcome was alert override rate. Secondary outcomes included the percentage of unique medication exposures avoided and predictors of alert override.

RESULTS

During the analysis period, 2817 unique alerts fired, representing 830 distinct patient-medication exposures for 577 unique patients. The overall alert override rate was 85% (80.3% for inpatient alerts and 95.8% for outpatient alerts). Of unique medication-patient exposures, 19% were avoided because of the alert. Assigned alert severity of "contraindicated" were less likely to be overridden (odds ratio [OR] 0.42, 95% confidence interval [CI] 0.32-0.56), as well as alerts activated during evening staffing (OR 0.69, 95% CI 0.55-0.87).

DISCUSSION

Implementation of a myasthenia gravis drug-disease interaction alert reduced overall patient exposure to potentially harmful medications by approximately 19%. Future optimization includes enhanced provider and pharmacist education. Further refinement of alert logic criteria to optimize medication risk reduction and reduce alert fatigue is warranted to support clinicians in prescribing and reduce electronic health record time burden.

Inpatient hospitalisation and mortality rate trends from 2004 to 2014 in the USA: a propensity score-matched case-control study of hyperkalaemia

OBJECTIVE

To study the trends of hyperkalaemia in USA inpatient hospitalisation records with heart failure (HF), chronic kidney disease (CKD), acute kidney injury (AKI) and/or type II diabetes mellitus (T2DM) from 2004 to 2014 with respect to prevalence and inpatient mortality.

DESIGN

Observational cross-sectional and propensity score-matched case-control study.

SETTING

The National Inpatient Sample (representing up to 97% of inpatient hospital discharge records in the USA) from 2004 to 2014 PARTICIPANTS: 120 513 483 (±2 312 391) adult inpatient hospitalisation records with HF, CKD/end-stage renal disease (ESRD), AKI and/or T2DM.

EXPOSURE

Hyperkalaemia, defined as the presence of an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) code of '276.7' in any of the first 15 diagnostic codes.

PRIMARY AND SECONDARY OUTCOME MEASURES

The outcomes of interest are the annual rates of hyperkalaemia prevalence and inpatient mortality.

RESULTS

Among 120 513 483 (±2 312 391) adult inpatient hospitalisations with HF, CKD/ESRD, AKI and/or T2DM, we found a 28.9% relative increase of hyperkalaemia prevalence from 4.94% in 2004 to 6.37% in 2014 (p<0.001). Hyperkalaemia was associated with an average of 4 percentage points higher rate of inpatient mortality (1.71 post-matching, p<0.0001). Inpatient mortality rates decreased from 11.49%±0.17% to 6.43%±0.08% and 9.67%±0.13% to 5.05%±0.07% for matched cases with and without hyperkalaemia, respectively (p<0.001).

CONCLUSIONS

Hyperkalaemia prevalence increased over time and was associated with greater inpatient mortality, even after accounting for presentation characteristics. We detected a decreasing trend in inpatient mortality risk, regardless of hyperkalaemia presence.

Modulators Influencing Medication Alert Acceptance: An Explorative Review

OBJECTIVES

Clinical decision support systems (CDSSs) use alerts to enhance medication safety and reduce medication error rates. A major challenge of medication alerts is their low acceptance rate, limiting their potential benefit. A structured overview about modulators influencing alert acceptance is lacking. Therefore, we aimed to review and compile qualitative and quantitative modulators of alert acceptance and organize them in a comprehensive model.

METHODS

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline, a literature search in PubMed was started in February 2018 and continued until October 2021. From all included articles, qualitative and quantitative parameters and their impact on alert acceptance were extracted. Related parameters were then grouped into factors, allocated to superordinate determinants, and subsequently further allocated into five categories that were already known to influence alert acceptance.

RESULTS

Out of 539 articles, 60 were included. A total of 391 single parameters were extracted (e.g., patients' comorbidity) and grouped into 75 factors (e.g., comorbidity), and 25 determinants (e.g., complexity) were consequently assigned to the predefined five categories, i.e., CDSS, care provider, patient, setting, and involved drug. More than half of all factors were qualitatively assessed (n = 21) or quantitatively inconclusive (n = 19). Furthermore, 33 quantitative factors clearly influenced alert acceptance (positive correlation: e.g., alert type, patients' comorbidity; negative correlation: e.g., number of alerts per care provider, moment of alert display in the workflow). Two factors (alert frequency, laboratory value) showed contradictory effects, meaning that acceptance was significantly influenced both positively and negatively by these factors, depending on the study. Interventional studies have been performed for only 12 factors while all other factors were evaluated descriptively.

CONCLUSION

This review compiles modulators of alert acceptance distinguished by being studied quantitatively or qualitatively and indicates their effect magnitude whenever possible. Additionally, it describes how further research should be designed to comprehensively quantify the effect of alert modulators.

Forecasting of Patient-Specific Kidney Transplant Function With a Sequence-to-Sequence Deep Learning Model

IMPORTANCE

Like other clinical biomarkers, trajectories of estimated glomerular filtration rate (eGFR) after kidney transplant are characterized by intra-individual variability. These fluctuations hamper the distinction between alarming graft functional deterioration or harmless fluctuation within the patient-specific expected reference range of eGFR.

OBJECTIVE

To determine whether a deep learning model could accurately predict the patient-specific expected reference range of eGFR after kidney transplant.

DESIGN, SETTING, AND PARTICIPANTS

A multicenter diagnostic study consisted of a derivation cohort of 933 patients who received a kidney transplant between 2004 and 2013 with 100 867 eGFR measurements from University Hospitals Leuven, Belgium, and 2 independent test cohorts: with 39 999 eGFR measurements from 1 170 patients, 1 from University Hospitals Leuven, Belgium, receiving transplants between 2013 and 2018 and 1 from Hannover Medical School, Germany, receiving transplants between 2003 and 2007. Patients receiving a single kidney transplant, with consecutive eGFR measurements were included. Data were analyzed from February 2019 to April 2021.

EXPOSURES

In the derivation cohort 100 867 eGFR measurements were available for analysis and 39 999 eGFR measurements from the independent test cohorts.

MAIN OUTCOMES AND MEASURES

A sequence-to-sequence model was developed for prediction of a patient-specific expected range of eGFR, based on previous eGFR values. The primary outcome was the performance of the deep learning sequence-to-sequence model in the 2 independent cohorts.

RESULTS

In this diagnostic study, a total of 933 patients in the training sets (mean [SD] age, 53.5 [13.3] years; 570 male [61.1%]) and 1170 patients in the independent test sets (cohort 1 [n = 621]: mean [SD] age, 58.5 [12.1] years; 400 male [64.4%]; cohort 2 [n = 549]: mean [SD] age, 50.1 [13.0] years; 316 male [57.6%]) who received a single kidney transplant most frequently from deceased donors, the sequence-to-sequence models accurately predicted future patient-specific eGFR trajectories within the first 3 months after transplant, based on the previous graft eGFR values (root mean square error, 6.4-8.9 mL/min/1.73 m2). The sequence-to-sequence model predictions outperformed the more conventional autoregressive integrated moving average prediction model, at all input/output number of eGFR values.

CONCLUSIONS AND RELEVANCE

In this diagnostic study, a sequence-to-sequence deep learning model was developed and validated for individual forecasting of kidney transplant function. The patient-specific sequence predictions could be used in clinical practice to guide physicians on deviations from the expected intra-individual variability, rather than relating the individual results to the reference range of the healthy population.

Systematic Quality Improvement and Metabolic Monitoring for Individuals Taking Antipsychotic Drugs

OBJECTIVE

The authors sought to increase the rate of cardiometabolic monitoring for patients receiving antipsychotic drugs in an academic outpatient psychiatric clinic serving people with serious mental illness.

METHODS

Using a prospective quasi-experimental, interrupted time-series design with data from the electronic health record (EHR), the authors determined metabolic monitoring rates before, during, and after implementation of prespecified quality improvement (QI) measures between August 2016 and July 2017. QI measures included a combination of provider, patient, and staff education; systematic barrier reduction; and an EHR-based reminder system.

RESULTS

After 1 year of QI implementation, the rate of metabolic monitoring had increased from 33% to 49% (p<0.01) for the primary outcome measure (hemoglobin A1C and lipid panel). This increased monitoring rate was sustained for 27 months beyond the end of the QI intervention. More than 75% of providers did not find the QI reminders burdensome.

CONCLUSIONS

Significant improvement in the rate of metabolic monitoring for people taking antipsychotic drugs can be achieved with little added burden on providers. Future research needs to assess the full range of patient, provider, and system barriers that prevent cardiometabolic monitoring for all individuals receiving antipsychotic drugs.

Redesign of computerized decision support system to improve Non Vitamin K oral anticoagulant prescribing-A pre and post qualitative and quantitative study

BACKGROUND

Inappropriate prescribing of non-vitamin K agents (NOAC) contributes to significant economic and personal burden to our society. Studies have shown that when well designed and targeted, computerized alerts can be effective in improving prescribing without contributing to alert fatigue.

METHOD

A collaborative multidisciplinary review group was set up to review and endorse an upgrade and modification to the hospital electronic medication management system (EMS). The intervention focused on implementing tailored electronic patient specific physiological alerts (such as age, renal function weight and drug interactions) built in EMS to improve the appropriateness of NOAC prescribing at this multisite teaching Australian hospital. To assess the qualitative and quantitative impact of the intervention, a pre and post retrospective study of NOAC prescribing of 100 patients' pre and post the implementation stage was conducted in a multisite Australian 650 bed hospital. Appropriateness of NOAC prescribing was assessed by an experienced pharmacist using approved prescribing product information recommendations. Prescriber satisfaction and experience survey was assessed in both stages of the study using a standard satisfaction survey. Associated hospital acquired complications (HAC) with potential inappropriate NOAC prescribing were evaluated as well as related admission cost and average length of stay.

RESULTS

Redesign of computerised decision support in EMS improved appropriateness of NOAC prescribing from 48 % to 91 %, P < 0.05. A total of 67 prescribers accepted the invitation to participate in the qualitative satisfaction study. Half the respondents (n = 33, 50 %) answered positively to a question assessing the usefulness of implementing NOAC alerts in the EMS in improving their practice and patient safety. This rate has increased to 72 % (n = 48) in the post intervention phase. P < 0.05. Additionally, the total number of reported HAC that are likely to be associated with inappropriate NOAC prescribing was reduced by 36 % in the post intervention phase (from 29 to 22 (RR = 0.7454 95 %CI (0.4283-1.2972), P = 0.2986). The cost of associated HAC has also reduced by 29 % (from $1,282,748 to $911,117) as well as the mean length stay by 11 % (from 18 days to 16 days) post the intervention phase.

CONCLUSION

This study highlights that well-designed electronic prescribing alerts that provide context-relevant information to prescribers are likely to result in benefits to clinicians and patients as well reduction in economic burden. Moreover, they could also contribute to reducing hospital acquired complications and lessen the economic burden on our society.

Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials

BACKGROUND

Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools.

OBJECTIVE

This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care.

METHODS

A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior.

RESULTS

To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images.

CONCLUSIONS

These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS.

TRIAL REGISTRATION

Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.

Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial

OBJECTIVE

To determine whether electronic health record alerts for acute kidney injury would improve patient outcomes of mortality, dialysis, and progression of acute kidney injury.

DESIGN

Double blinded, multicenter, parallel, randomized controlled trial.

SETTING

Six hospitals (four teaching and two non-teaching) in the Yale New Haven Health System in Connecticut and Rhode Island, US, ranging from small community hospitals to large tertiary care centers.

PARTICIPANTS

6030 adult inpatients with acute kidney injury, as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) creatinine criteria.

INTERVENTIONS

An electronic health record based "pop-up" alert for acute kidney injury with an associated acute kidney injury order set upon provider opening of the patient's medical record.

MAIN OUTCOME MEASURES

A composite of progression of acute kidney injury, receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included outcomes at each hospital and frequency of various care practices for acute kidney injury.

RESULTS

6030 patients were randomized over 22 months. The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67). Analysis by each hospital showed worse outcomes in the two non-teaching hospitals (n=765, 13%), where alerts were associated with a higher risk of the primary outcome (relative risk 1.49, 95% confidence interval 1.12 to 1.98, P=0.006). More deaths occurred at these centers (15.6% in the alert group v 8.6% in the usual care group, P=0.003). Certain acute kidney injury care practices were increased in the alert group but did not appear to mediate these outcomes.

CONCLUSIONS

Alerts did not reduce the risk of our primary outcome among patients in hospital with acute kidney injury. The heterogeneity of effect across clinical centers should lead to a re-evaluation of existing alerting systems for acute kidney injury.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02753751.

Alarm Fatigue in the Intensive Care Unit: Relevance and Response Time

OBJECTIVE

To establish the presence of alarm fatigue, the clinical relevance of alarms and the stimulus-response time of the health team in an Adult Intensive Care Unit.

METHOD

Descriptive, quantitative, observational study, developed in the Multipurpose Adult Intensive Care Unit. Population made up of health personnel and the ICU teams. The method used was non-participant observation. Follow-up was carried out over 120 hours in three months. The variables studied were number of alarms activated, time elapsed between the alert sound of the blood pressure parameter, heart rate and oximetry and the response of the health personnel who attended the alarm. A descriptive statistical analysis was carried out.

RESULTS

5,147 alarms were detected, on average 43 alarms / hour, of these 52.8% corresponded to multiparameter monitors and the rest to other equipment. Of those generated by multiparameter monitors, 37.3% were blood pressure, 33.4% oximetry and 29.3% heart rate. The clinical relevance was low in 42.7%, medium in 49.8% and high in 7.5%. The stimulus response time was between 0 and 60 seconds for 37% of the alarms; however, 42.5% had no response, which is why they are considered fatigued. A statistically significant relationship was found between the response time and the clinical relevance of the alarms (p = .000).

CONCLUSIONS

The presence of alarm fatigue was evident; with predominance of clinical relevance in the middle and low ranges. The health personnel responded within the time established for timely attention to the non-fatigued alarms.

Adaptive design of a clinical decision support tool: What the impact on utilization rates means for future CDS research

OBJECTIVE

We employed an agile, user-centered approach to the design of a clinical decision support tool in our prior integrated clinical prediction rule study, which achieved high adoption rates. To understand if applying this user-centered process to adapt clinical decision support tools is effective in improving the use of clinical prediction rules, we examined utilization rates of a clinical decision support tool adapted from the original integrated clinical prediction rule study tool to determine if applying this user-centered process to design yields enhanced utilization rates similar to the integrated clinical prediction rule study.

MATERIALS & METHODS: We conducted pre-deployment usability testing and semi-structured group interviews at 6 months post-deployment with 75 providers at 14 intervention clinics across the two sites to collect user feedback. Qualitative data analysis is bifurcated into immediate and delayed stages; we reported on immediate-stage findings from real-time field notes used to generate a set of rapid, pragmatic recommendations for iterative refinement. Monthly utilization rates were calculated and examined over 12 months.

RESULTS

We hypothesized a well-validated, user-centered clinical decision support tool would lead to relatively high adoption rates. Then 6 months post-deployment, integrated clinical prediction rule study tool utilization rates were substantially lower than anticipated based on the original integrated clinical prediction rule study trial (68%) at 17% (Health System A) and 5% (Health System B). User feedback at 6 months resulted in recommendations for tool refinement, which were incorporated when possible into tool design; however, utilization rates at 12 months post-deployment remained low at 14% and 4% respectively.

DISCUSSION

Although valuable, findings demonstrate the limitations of a user-centered approach given the complexity of clinical decision support.

CONCLUSION

Strategies for addressing persistent external factors impacting clinical decision support adoption should be considered in addition to the user-centered design and implementation of clinical decision support.

Integrating Palliative Care Services in Ambulatory Oncology: An Application of the Edmonton Symptom Assessment System

PURPOSE

Research in palliative care demonstrates improvements in overall survival, quality of life, symptom management, and reductions in the cost of care. Despite the American Society of Clinical Oncology recommendation for early concurrent palliative care in patients with advanced cancer and high symptom burden, integrating palliative services is challenging. Our aims were to quantitatively describe the palliative referral rates and symptom burden in a South Texas cancer center and establish a palliative referral system by implementing the Edmonton Symptom Assessment Scale (ESAS).

METHODS

As part of our Plan-Do-Study-Act process, all staff received an educational overview of the ESAS tool and consultation ordering process. The ESAS form was then implemented across five ambulatory oncology clinics to assess symptom burden and changes therein longitudinally. Referral rates and symptom assessment scores were tracked as metrics for quality improvement.

RESULTS

On average, one patient per month was referred before implementation of the intervention compared with 10 patients per month after implementation across all clinics. In five sample clinics, 607 patients completed the initial assessment, and 430 follow-up forms were collected over 5 months, resulting in a total of 1,037 scores collected in REDCap. The mean ESAS score for initial patient visits was 20.0 (standard deviation, 18.1), and referred patients had an initial mean score of 39.0 (standard deviation, 19.0).

CONCLUSION

This project highlights the low palliative care consultation rate, high symptom burden of oncology patients, and underuse of services by oncologists despite improvements with the introduction of a symptom assessment form and referral system.