The design of studies testing the effectiveness of risk-guided care has many challenges: a scoping review addressing key considerations

OBJECTIVES

Studies evaluating the effectiveness of care based on patients' risk of adverse outcomes (risk-guided care) use a variety of study designs. In this scoping review, using examples, we review characteristics of relevant studies and present key design features to optimize the trustworthiness of results.

STUDY DESIGN AND SETTING

We searched five online databases for studies evaluating the effect of risk-guided care among adults on clinical outcomes, process, or cost. Pairs of reviewers independently performed screening and data abstraction. We descriptively summarized the study design and characteristics.

RESULTS

Among 14,561 hits, we identified 116 eligible studies. Study designs included randomized controlled trials (RCTs), post hoc analysis of RCTs, and retrospective or prospective cohort studies. Challenges and sources of bias in the design included limited performance of predictive models, contamination, inadequacy to address the credibility of subgroup effects, absence of differences in care across risk strata, reporting only process measures as opposed to clinical outcomes, and failure to report benefits and harms.

CONCLUSION

To assess the benefit of risk-guided care, RCTs provide the most trustworthy evidence. Observational studies offer an alternative but are hampered by confounding and other limitations. Reaching valid conclusions when testing risk-guided care requires addressing the challenges identified in our review.

Reducing prescribing of antibiotics for acute respiratory infections using a frontline nurse-led EHR-Integrated clinical decision support tool: protocol for a stepped wedge randomized control trial

BACKGROUND

Overprescribing of antibiotics for acute respiratory infections (ARIs) remains a major issue in outpatient settings. Use of clinical prediction rules (CPRs) can reduce inappropriate antibiotic prescribing but they remain underutilized by physicians and advanced practice providers. A registered nurse (RN)-led model of an electronic health record-integrated CPR (iCPR) for low-acuity ARIs may be an effective alternative to address the barriers to a physician-driven model.

METHODS

Following qualitative usability testing, we will conduct a stepped-wedge practice-level cluster randomized controlled trial (RCT) examining the effect of iCPR-guided RN care for low acuity patients with ARI. The primary hypothesis to be tested is: Implementation of RN-led iCPR tools will reduce antibiotic prescribing across diverse primary care settings. Specifically, this study aims to: (1) determine the impact of iCPRs on rapid strep test and chest x-ray ordering and antibiotic prescribing rates when used by RNs; (2) examine resource use patterns and cost-effectiveness of RN visits across diverse clinical settings; (3) determine the impact of iCPR-guided care on patient satisfaction; and (4) ascertain the effect of the intervention on RN and physician burnout.

DISCUSSION

This study represents an innovative approach to using an iCPR model led by RNs and specifically designed to address inappropriate antibiotic prescribing. This study has the potential to provide guidance on the effectiveness of delegating care of low-acuity patients with ARIs to RNs to increase use of iCPRs and reduce antibiotic overprescribing for ARIs in outpatient settings.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT04255303, Registered February 5 2020, https://clinicaltrials.gov/ct2/show/NCT04255303 .

Integrating Clinical Decision Support Into Electronic Health Record Systems Using a Novel Platform (EvidencePoint): Developmental Study

BACKGROUND

Through our work, we have demonstrated how clinical decision support (CDS) tools integrated into the electronic health record (EHR) assist providers in adopting evidence-based practices. This requires confronting technical challenges that result from relying on the EHR as the foundation for tool development; for example, the individual CDS tools need to be built independently for each different EHR.

OBJECTIVE

The objective of our research was to build and implement an EHR-agnostic platform for integrating CDS tools, which would remove the technical constraints inherent in relying on the EHR as the foundation and enable a single set of CDS tools that can work with any EHR.

METHODS

We developed EvidencePoint, a novel, cloud-based, EHR-agnostic CDS platform, and we will describe the development of EvidencePoint and the deployment of its initial CDS tools, which include EHR-integrated applications for clinical use cases such as prediction of hospitalization survival for patients with COVID-19, venous thromboembolism prophylaxis, and pulmonary embolism diagnosis.

RESULTS

The results below highlight the adoption of the CDS tools, the International Medical Prevention Registry on Venous Thromboembolism-D-Dimer, the Wells' criteria, and the Northwell COVID-19 Survival (NOCOS), following development, usability testing, and implementation. The International Medical Prevention Registry on Venous Thromboembolism-D-Dimer CDS was used in 5249 patients at the 2 clinical intervention sites. The intervention group tool adoption was 77.8% (4083/5249 possible uses). For the NOCOS tool, which was designed to assist with triaging patients with COVID-19 for hospital admission in the event of constrained hospital resources, the worst-case resourcing scenario never materialized and triaging was never required. As a result, the NOCOS tool was not frequently used, though the EvidencePoint platform's flexibility and customizability enabled the tool to be developed and deployed rapidly under the emergency conditions of the pandemic. Adoption rates for the Wells' criteria tool will be reported in a future publication.

CONCLUSIONS

The EvidencePoint system successfully demonstrated that a flexible, user-friendly platform for hosting CDS tools outside of a specific EHR is feasible. The forthcoming results of our outcomes analyses will demonstrate the adoption rate of EvidencePoint tools as well as the impact of behavioral economics "nudges" on the adoption rate. Due to the EHR-agnostic nature of EvidencePoint, the development process for additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system of EvidencePoint.

Disseminating child abuse clinical decision support among commercial electronic health records: Effects on clinical practice

Objectives

The use of electronic health record (EHR)-embedded child abuse clinical decision support (CA-CDS) may help decrease morbidity from child maltreatment. We previously reported on the development of CA-CDS in Epic and Allscripts. The objective of this study was to implement CA-CDS into Epic and Allscripts and determine its effects on identification, evaluation, and reporting of suspected child maltreatment.

Materials and Methods

After a preimplementation period, CA-CDS was implemented at University of Wisconsin (Epic) and Northwell Health (Allscripts). Providers were surveyed before the go-live and 4 months later. Outcomes included the proportion of children who triggered the CA-CDS system, had a positive Child Abuse Screen (CAS) and/or were reported to Child Protective Services (CPS).

Results

At University of Wisconsin (UW), 3.5% of children in the implementation period triggered the system. The CAS was positive in 1.8% of children. The proportion of children reported to CPS increased from 0.6% to 0.9%. There was rapid uptake of the abuse order set.At Northwell Health (NW), 1.9% of children in the implementation period triggered the system. The CAS was positive in 1% of children. The child abuse order set was rarely used. Preimplementation, providers at both sites were similar in desire to have CA-CDS system and perception of CDS in general. After implementation, UW providers had a positive perception of the CA-CDS system, while NW providers had a negative perception.

Discussion

CA-CDS was able to be implemented in 2 different EHRs with differing effects on clinical care and provider feedback. At UW, the site with higher uptake of the CA-CDS system, the proportion of children who triggered the system and the rate of positive CAS was similar to previous studies and there was an increase in the proportion of cases of suspected abuse identified as measured by reports to CPS. Our data demonstrate how local environment, end-users' opinions, and limitations in the EHR platform can impact the success of implementation.

Conclusions

When disseminating CA-CDS into different hospital systems and different EHRs, it is critical to recognize how limitations in the functionality of the EHR can impact the success of implementation. The importance of collecting, interpreting, and responding to provider feedback is of critical importance particularly with CDS related to child maltreatment.

Nudging Health Care Providers' Adoption of Clinical Decision Support: Protocol for the User-Centered Development of a Behavioral Economics-Inspired Electronic Health Record Tool

BACKGROUND

The improvements in care resulting from clinical decision support (CDS) have been significantly limited by consistently low health care provider adoption. Health care provider attitudes toward CDS, specifically psychological and behavioral barriers, are not typically addressed during any stage of CDS development, although they represent an important barrier to adoption. Emerging evidence has shown the surprising power of using insights from the field of behavioral economics to address psychological and behavioral barriers. Nudges are formal applications of behavioral economics, defined as positive reinforcement and indirect suggestions that have a nonforced effect on decision-making.

OBJECTIVE

Our goal is to employ a user-centered design process to develop a CDS tool-the pulmonary embolism (PE) risk calculator-for PE risk stratification in the emergency department that incorporates a behavior theory-informed nudge to address identified behavioral barriers to use.

METHODS

All study activities took place at a large academic health system in the New York City metropolitan area. Our study used a user-centered and behavior theory-based approach to achieve the following two aims: (1) use mixed methods to identify health care provider barriers to the use of an active CDS tool for PE risk stratification and (2) develop a new CDS tool-the PE risk calculator-that addresses behavioral barriers to health care providers' adoption of CDS by incorporating nudges into the user interface. These aims were guided by the revised Observational Research Behavioral Information Technology model. A total of 50 clinicians who used the original version of the tool were surveyed with a quantitative instrument that we developed based on a behavior theory framework-the Capability-Opportunity-Motivation-Behavior framework. A semistructured interview guide was developed based on the survey responses. Inductive methods were used to analyze interview session notes and audio recordings from 12 interviews. Revised versions of the tool were developed that incorporated nudges.

RESULTS

Functional prototypes were developed by using Axure PRO (Axure Software Solutions) software and usability tested with end users in an iterative agile process (n=10). The tool was redesigned to address 4 identified major barriers to tool use; we included 2 nudges and a default. The 6-month pilot trial for the tool was launched on October 1, 2021.

CONCLUSIONS

Clinicians highlighted several important psychological and behavioral barriers to CDS use. Addressing these barriers, along with conducting traditional usability testing, facilitated the development of a tool with greater potential to transform clinical care. The tool will be tested in a prospective pilot trial.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/42653.

Automated Pulmonary Embolism Risk Assessment Using the Wells Criteria: Validation Study

Background

Computed tomography pulmonary angiography (CTPA) is frequently used in the emergency department (ED) for the diagnosis of pulmonary embolism (PE), while posing risk for contrast-induced nephropathy and radiation-induced malignancy.

Objective

We aimed to create an automated process to calculate the Wells score for pulmonary embolism for patients in the ED, which could potentially reduce unnecessary CTPA testing.

Methods

We designed an automated process using electronic health records data elements, including using a combinatorial keyword search method to query free-text fields, and calculated automated Wells scores for a sample of all adult ED encounters that resulted in a CTPA study for PE at 2 tertiary care hospitals in New York, over a 2-month period. To validate the automated process, the scores were compared to those derived from a 2-clinician chart review.

Results

A total of 202 ED encounters resulted in a completed CTPA to form the retrospective study cohort. Patients classified as “PE likely” by the automated process (126/202, 62%) had a PE prevalence of 15.9%, whereas those classified as “PE unlikely” (76/202, 38%; Wells score >4) had a PE prevalence of 7.9%. With respect to classification of the patient as “PE likely,” the automated process achieved an accuracy of 92.1% when compared with the chart review, with sensitivity, specificity, positive predictive value, and negative predictive value of 93%, 90.5%, 94.4%, and 88.2%, respectively.

Conclusions

This was a successful development and validation of an automated process using electronic health records data elements, including free-text fields, to classify risk for PE in ED visits.

Pulse-shape discrimination against low-energy Ar-39 beta decays in liquid argon with 4.5 tonne-years of DEAP-3600 data

The DEAP-3600 detector searches for the scintillation signal from dark matter particles scattering on a 3.3 tonne liquid argon target. The largest background comes from 39 Ar beta decays and is suppressed using pulse-shape discrimination (PSD). We use two types of PSD estimator: the prompt-fraction, which considers the fraction of the scintillation signal in a narrow and a wide time window around the event peak, and the log-likelihood-ratio, which compares the observed photon arrival times to a signal and a background model. We furthermore use two algorithms to determine the number of photons detected at a given time: (1) simply dividing the charge of each PMT pulse by the mean single-photoelectron charge, and (2) a likelihood analysis that considers the probability to detect a certain number of photons at a given time, based on a model for the scintillation pulse shape and for afterpulsing in the light detectors. The prompt-fraction performs approximately as well as the log-likelihood-ratio PSD algorithm if the photon detection times are not biased by detector effects. We explain this result using a model for the information carried by scintillation photons as a function of the time when they are detected.

Barriers to the Use of Clinical Decision Support for the Evaluation of Pulmonary Embolism: Qualitative Interview Study

BACKGROUND

Clinicians often disregard potentially beneficial clinical decision support (CDS).

OBJECTIVE

In this study, we sought to explore the psychological and behavioral barriers to the use of a CDS tool.

METHODS

We conducted a qualitative study involving emergency medicine physicians and physician assistants. A semistructured interview guide was created based on the Capability, Opportunity, and Motivation-Behavior model. Interviews focused on the barriers to the use of a CDS tool built based on Wells' criteria for pulmonary embolism to assist clinicians in establishing pretest probability of pulmonary embolism before imaging.

RESULTS

Interviews were conducted with 12 clinicians. Six barriers were identified, including (1) Bayesian reasoning, (2) fear of missing a pulmonary embolism, (3) time pressure or cognitive load, (4) gestalt includes Wells' criteria, (5) missed risk factors, and (6) social pressure.

CONCLUSIONS

Clinicians highlighted several important psychological and behavioral barriers to CDS use. Addressing these barriers will be paramount in developing CDS that can meet its potential to transform clinical care.

External validation of the IMPROVE-DD risk assessment model for venous thromboembolism among inpatients with COVID-19

There is a need to discriminate which COVID-19 inpatients are at higher risk for venous thromboembolism (VTE) to inform prophylaxis strategies. The IMPROVE-DD VTE risk assessment model (RAM) has previously demonstrated good discrimination in non-COVID populations. We aimed to externally validate the IMPROVE-DD VTE RAM in medical patients hospitalized with COVID-19. This retrospective cohort study evaluated the IMPROVE-DD VTE RAM in adult patients with COVID-19 admitted to one of thirteen Northwell Health hospitals in the New York metropolitan area between March 1, 2020 and April 27, 2020. VTE was defined as new-onset symptomatic deep venous thrombosis or pulmonary embolism. To assess the predictive value of the RAM, the receiver operating characteristic (ROC) curve was plotted and the area under the curve (AUC) was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Of 9407 patients who met study criteria, 274 patients developed VTE with a prevalence of 2.91%. The VTE rate was 0.41% for IMPROVE-DD score 0–1 (low risk), 1.21% for score 2–3 (moderate risk), and 5.30% for score ≥ 4 (high risk). Approximately 45.7% of patients were classified as high VTE risk, 33.3% moderate risk, and 21.0% low risk. Discrimination of low versus moderate-high VTE risk demonstrated sensitivity 0.971, specificity 0.215, PPV 0.036, and NPV 0.996. ROC AUC was 0.703. In this external validation study, the IMPROVE-DD VTE RAM demonstrated very good discrimination to identify hospitalized COVID-19 patients at low, moderate, and high VTE risk.

In-Hospital 30-Day Survival Among Young Adults With Coronavirus Disease 2019: A Cohort Study

BACKGROUND

Our objective was to characterize young adult patients hospitalized with coronavirus disease 2019 (COVID-19) and identify predictors of survival at 30 days.

METHODS

This retrospective cohort study took place at 12 acute care hospitals in the New York City area. Patients aged 18-39 hospitalized with confirmed COVID-19 between March 1 and April 27, 2020 were included in the study. Demographic, clinical, and outcome data were extracted from electronic health record reports.

RESULTS

A total of 1013 patients were included in the study (median age, 33 years; interquartile range [IQR], 28-36; 52% female). At the study end point, 940 (92.8%) patients were discharged alive, 18 (1.8%) remained hospitalized, 5 (0.5%) were transferred to another acute care facility, and 50 (4.9%) died. The most common comorbidities in hospitalized young adult patients were obesity (51.2%), diabetes mellitus (14.8%), and hypertension (13%). Multivariable analysis revealed that obesity (adjusted hazard ratio [aHR], 2.71; 95% confidence interval [CI], 1.28-5.73; P = .002) and Charlson comorbidity index score (aHR, 1.20; 95% CI, 1.07-1.35; P = .002) were independent predictors of in-hospital 30-day mortality.

CONCLUSIONS

Obesity was identified as the strongest negative predictor of 30-day in-hospital survival in young adults with COVID-19.

A Machine Learning Prediction Model of Respiratory Failure Within 48 Hours of Patient Admission for COVID-19: Model Development and Validation

BACKGROUND

Predicting early respiratory failure due to COVID-19 can help triage patients to higher levels of care, allocate scarce resources, and reduce morbidity and mortality by appropriately monitoring and treating the patients at greatest risk for deterioration. Given the complexity of COVID-19, machine learning approaches may support clinical decision making for patients with this disease.

OBJECTIVE

Our objective is to derive a machine learning model that predicts respiratory failure within 48 hours of admission based on data from the emergency department.

METHODS

Data were collected from patients with COVID-19 who were admitted to Northwell Health acute care hospitals and were discharged, died, or spent a minimum of 48 hours in the hospital between March 1 and May 11, 2020. Of 11,525 patients, 933 (8.1%) were placed on invasive mechanical ventilation within 48 hours of admission. Variables used by the models included clinical and laboratory data commonly collected in the emergency department. We trained and validated three predictive models (two based on XGBoost and one that used logistic regression) using cross-hospital validation. We compared model performance among all three models as well as an established early warning score (Modified Early Warning Score) using receiver operating characteristic curves, precision-recall curves, and other metrics.

RESULTS

The XGBoost model had the highest mean accuracy (0.919; area under the curve=0.77), outperforming the other two models as well as the Modified Early Warning Score. Important predictor variables included the type of oxygen delivery used in the emergency department, patient age, Emergency Severity Index level, respiratory rate, serum lactate, and demographic characteristics.

CONCLUSIONS

The XGBoost model had high predictive accuracy, outperforming other early warning scores. The clinical plausibility and predictive ability of XGBoost suggest that the model could be used to predict 48-hour respiratory failure in admitted patients with COVID-19.

Validation of the IMPROVE-DD risk assessment model for venous thromboembolism among hospitalized patients with COVID-19

BACKGROUND

Antithrombotic guidance statements for hospitalized patients with coronavirus disease 2019 (COVID-19) suggest a universal thromboprophylactic strategy with potential to escalate doses in high-risk patients. To date, no clear approach exists to discriminate patients at high risk for venous thromboembolism (VTE).

OBJECTIVES

The objective of this study is to externally validate the IMPROVE-DD risk assessment model (RAM) for VTE in a large cohort of hospitalized patients with COVID-19 within a multihospital health system.

METHODS

This retrospective cohort study evaluated the IMPROVE-DD RAM on adult inpatients with COVID-19 hospitalized between March 1, 2020, and April 27, 2020. Diagnosis of VTE was defined by new acute deep venous thrombosis or pulmonary embolism by Radiology Department imaging or point-of-care ultrasound. The receiver operating characteristic (ROC) curve was plotted and area under the curve (AUC) calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using standard methods.

RESULTS

A total of 9407 patients were included, with a VTE prevalence of 2.9%. The VTE rate was 0.4% for IMPROVE-DD score 0-1 (low risk), 1.3% for score 2-3 (moderate risk), and 5.3% for score ≥ 4 (high risk). Approximately 45% of the total population scored high VTE risk, while 21% scored low VTE risk. IMPROVE-DD discrimination of low versus medium/high risk showed sensitivity of 0.971, specificity of 0.218, PPV of 0.036, and NPV of 0.996. ROC AUC was 0.702.

CONCLUSIONS

The IMPROVE-DD VTE RAM demonstrated very good discrimination to identify hospitalized patients with COVID-19 as low, moderate, and high VTE risk in this large external validation study with potential to individualize thromboprophylactic strategies.

Dissemination of child abuse clinical decision support: Moving beyond a single electronic health record

BACKGROUND

Child maltreatment is a leading cause of pediatric morbidity and mortality. We previously reported on development and implementation of a child abuse clinical decision support system (CA-CDSS) in the Cerner electronic health record (EHR). Our objective was to develop a CA-CDSS in two different EHRs.

METHODS

Using the CA-CDSS in Cerner as a template, CA-CDSSs were developed for use in four hospitals in the Northwell Health system who use Allscripts and two hospitals in the University of Wisconsin health system who use Epic. Each system had a combination of triggers, alerts and child abuse-specific order sets. Usability evaluation was done prior to launch of the CA-CDSS.

RESULTS

Over an 18-month period, a CA-CDSS was embedded into Epic and Allscripts at two hospital systems. The CA-CDSSs vary significantly from each other in terms of the type of triggers which were able to be used, the type of alert, the ability of the alert to link directly to child abuse-specific order sets and the order sets themselves.

CONCLUSIONS

Dissemination of CA-CDSS from one EHR into the EHR in other health care systems is possible but time-consuming and needs to be adapted to the strengths and limitations of the specific EHR. Site-specific usability evaluation, buy-in of multiple stakeholder groups and significant information technology support are needed. These barriers limit scalability and widespread dissemination of CA-CDSS.

Let Sleeping Patients Lie, avoiding unnecessary overnight vitals monitoring using a clinically based deep-learning model

Impaired sleep for hospital patients is an all too common reality. Sleep disruptions due to unnecessary overnight vital sign monitoring are associated with delirium, cognitive impairment, weakened immunity, hypertension, increased stress, and mortality. It is also one of the most common complaints of hospital patients while imposing additional burdens on healthcare providers. Previous efforts to forgo overnight vital sign measurements and improve patient sleep used providers’ subjective stability assessment or utilized an expanded, thus harder to retrieve, set of vitals and laboratory results to predict overnight clinical risk. Here, we present a model that incorporates past values of a small set of vital signs and predicts overnight stability for any given patient-night. Using data obtained from a multi-hospital health system between 2012 and 2019, a recurrent deep neural network was trained and evaluated using ~2.3 million admissions and 26 million vital sign assessments. The algorithm is agnostic to patient location, condition, and demographics, and relies only on sequences of five vital sign measurements, a calculated Modified Early Warning Score, and patient age. We achieved an area under the receiver operating characteristic curve of 0.966 (95% confidence interval [CI] 0.956–0.967) on the retrospective testing set, and 0.971 (95% CI 0.965–0.974) on the prospective set to predict overnight patient stability. The model enables safe avoidance of overnight monitoring for ~50% of patient-nights, while only misclassifying 2 out of 10,000 patient-nights as stable. Our approach is straightforward to deploy, only requires regularly obtained vital signs, and delivers easily actionable clinical predictions for a peaceful sleep in hospitals.

Impact of Clinical Decision Support on Antibiotic Prescribing for Acute Respiratory Infections: a Cluster Randomized Implementation Trial

BACKGROUND

Clinical decision support (CDS) is a promising tool for reducing antibiotic prescribing for acute respiratory infections (ARIs).

OBJECTIVE

To assess the impact of previously effective CDS on antibiotic-prescribing rates for ARIs when adapted and implemented in diverse primary care settings.

DESIGN

Cluster randomized clinical trial (RCT) implementing a CDS tool designed to guide evidence-based evaluation and treatment of streptococcal pharyngitis and pneumonia.

SETTING

Two large academic health system primary care networks with a mix of providers.

PARTICIPANTS

All primary care practices within each health system were invited. All providers within participating clinic were considered a participant. Practices were randomized selection to a control or intervention group.

INTERVENTIONS

Intervention practice providers had access to an integrated clinical prediction rule (iCPR) system designed to determine the risk of bacterial infection from reason for visit of sore throat, cough, or upper respiratory infection and guide evidence-based evaluation and treatment.

MAIN OUTCOME(S)

Change in overall antibiotic prescription rates.

MEASURE(S)

Frequency, rates, and type of antibiotics prescribed in intervention and controls groups.

RESULTS

33 primary care practices participated with 541 providers and 100,573 patient visits. Intervention providers completed the tool in 6.9% of eligible visits. Antibiotics were prescribed in 35% and 36% of intervention and control visits, respectively, showing no statistically significant difference. There were also no differences in rates of orders for rapid streptococcal tests (RR, 0.94; P = 0.11) or chest X-rays (RR, 1.01; P = 0.999) between groups.

CONCLUSIONS

The iCPR tool was not effective in reducing antibiotic prescription rates for upper respiratory infections in diverse primary care settings. This has implications for the generalizability of CDS tools as they are adapted to heterogeneous clinical contexts.

TRIAL REGISTRATION

Clinicaltrials.gov (NCT02534987). Registered August 26, 2015 at https://clinicaltrials.gov.

“Ambulatory Management of Moderate to High Risk COVID-19 Patients: The Coronavirus Related Outpatient Work Navigators (CROWN) Protocol”

During the height of the novel 2019 coronavirus disease (COVID-19) pandemic in New York City, area hospitals were filled to 150% capacity, and there was a significant fear among the public of going to the hospital. Many hospitalized patients were treated with therapies that could be administered in a home setting under proper monitoring. We designed the CROWN Program, a Home-Care based ambulatory protocol to evaluate, monitor, and treat moderate to high risk COVID-19 patients in their homes, with escalation to hospital care when necessary. Patients were evaluated with telehealth visits with a Pulmonologist, and a Home-Care protocol, including RN visit, pulse-oximetry, and oxygen, lab-work, intravenous fluids, medication if needed patient data, comorbidities, and symptoms were collected. Labs, including COVID-19 PCR, D Dimer, CRP, Ferritin, Procalcitonin, CBC, and metabolic panel were measured, as were homecare, home oxygen, and intravenous fluids orders, radiographic studies and initiation of an anticoagulant. Emergency Department visits and need for hospital admission during the study period were recorded. A total of 182 patients were enrolled between the start date of April 27th and June 1st, and fell into two categories: not-admitted (101) and post-discharge (81). Two patients were referred for hospital admission, seven were treated and released from the ED, and one was referred to home hospice. There were no unexpected admissions or deaths. The CROWN program has demonstrated the feasibility and apparent safety of a specialized, Home-Care based protocol for the ambulatory management of moderate to high risk COVID-19 patients.

Prevalence of SARS-CoV-2 Antibodies in Health Care Personnel in the New York City Area

This seroprevalence survey study describes severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) seropositivity among health care workers at a New York City–based health system by age, sex, race, county of residence, and prior PCR-confirmed viral exposure.

Development and Validation of a Survival Calculator for Hospitalized Patients with COVID-19

BACKGROUND

Chinese studies reported predictors of severe disease and mortality associated with coronavirus disease 2019 (COVID-19). A generalizable and simple survival calculator based on data from US patients hospitalized with COVID-19 has not yet been introduced.

OBJECTIVE

Develop and validate a clinical tool to predict 7-day survival in patients hospitalized with COVID-19.

DESIGN

Retrospective and prospective cohort study.

SETTING

Thirteen acute care hospitals in the New York City area.

PARTICIPANTS

Adult patients hospitalized with a confirmed diagnosis of COVID-19. The development and internal validation cohort included patients hospitalized between March 1 and May 6, 2020. The external validation cohort included patients hospitalized between March 1 and May 5, 2020.

MEASUREMENTS

Demographic, laboratory, clinical, and outcome data were extracted from the electronic health record. Optimal predictors and performance were identified using least absolute shrinkage and selection operator (LASSO) regression with receiver operating characteristic curves and measurements of area under the curve (AUC).

RESULTS

The development and internal validation cohort included 11 095 patients with a median age of 65 years [interquartile range (IQR) 54-77]. Overall 7-day survival was 89%. Serum blood urea nitrogen, age, absolute neutrophil count, red cell distribution width, oxygen saturation, and serum sodium were identified as the 6 optimal of 42 possible predictors of survival. These factors constitute the NOCOS (Northwell COVID-19 Survival) Calculator. Performance in the internal validation, prospective validation, and external validation were marked by AUCs of 0.86, 0.82, and 0.82, respectively.

LIMITATIONS

All participants were hospitalized within the New York City area.

CONCLUSIONS

The NOCOS Calculator uses 6 factors routinely available at hospital admission to predict 7-day survival for patients hospitalized with COVID-19. The calculator is publicly available at https://feinstein.northwell.edu/NOCOS.

Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized With COVID-19 in the New York City Area

IMPORTANCE

There is limited information describing the presenting characteristics and outcomes of US patients requiring hospitalization for coronavirus disease 2019 (COVID-19).

OBJECTIVE

To describe the clinical characteristics and outcomes of patients with COVID-19 hospitalized in a US health care system.

DESIGN, SETTING, AND PARTICIPANTS

Case series of patients with COVID-19 admitted to 12 hospitals in New York City, Long Island, and Westchester County, New York, within the Northwell Health system. The study included all sequentially hospitalized patients between March 1, 2020, and April 4, 2020, inclusive of these dates.

EXPOSURES

Confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection by positive result on polymerase chain reaction testing of a nasopharyngeal sample among patients requiring admission.

MAIN OUTCOMES AND MEASURES

Clinical outcomes during hospitalization, such as invasive mechanical ventilation, kidney replacement therapy, and death. Demographics, baseline comorbidities, presenting vital signs, and test results were also collected.

RESULTS

A total of 5700 patients were included (median age, 63 years [interquartile range {IQR}, 52-75; range, 0-107 years]; 39.7% female). The most common comorbidities were hypertension (3026; 56.6%), obesity (1737; 41.7%), and diabetes (1808; 33.8%). At triage, 30.7% of patients were febrile, 17.3% had a respiratory rate greater than 24 breaths/min, and 27.8% received supplemental oxygen. The rate of respiratory virus co-infection was 2.1%. Outcomes were assessed for 2634 patients who were discharged or had died at the study end point. During hospitalization, 373 patients (14.2%) (median age, 68 years [IQR, 56-78]; 33.5% female) were treated in the intensive care unit care, 320 (12.2%) received invasive mechanical ventilation, 81 (3.2%) were treated with kidney replacement therapy, and 553 (21%) died. As of April 4, 2020, for patients requiring mechanical ventilation (n = 1151, 20.2%), 38 (3.3%) were discharged alive, 282 (24.5%) died, and 831 (72.2%) remained in hospital. The median postdischarge follow-up time was 4.4 days (IQR, 2.2-9.3). A total of 45 patients (2.2%) were readmitted during the study period. The median time to readmission was 3 days (IQR, 1.0-4.5) for readmitted patients. Among the 3066 patients who remained hospitalized at the final study follow-up date (median age, 65 years [IQR, 54-75]), the median follow-up at time of censoring was 4.5 days (IQR, 2.4-8.1).

CONCLUSIONS AND RELEVANCE

This case series provides characteristics and early outcomes of sequentially hospitalized patients with confirmed COVID-19 in the New York City area.

Comparison of international societal guidelines for the diagnosis of suspected pulmonary embolism during pregnancy

Pregnancy-associated pulmonary embolism is one of the leading causes of maternal mortality. Diagnosis of pulmonary embolism in pregnancy is challenging, with symptoms of pulmonary embolism mimicking those of pregnancy. Several key components such as clinical prediction tools, risk stratification, laboratory tests, and imaging widely used for diagnosis of pulmonary embolism in the non-pregnant population show limitations for diagnosis in pregnancy. Further, because of the difficulty of studying pregnant patients, high-quality research evaluating the performance of these diagnostic components in pregnancy is scarce. Seven international medical society guidelines present clinical diagnostic pathways for evaluation of pulmonary embolism in pregnancy that show conflicting recommendations on the use of these diagnostic components. This Review assesses all key components of diagnostic clinical pathways recommended by guidelines for evaluation of pulmonary embolism in pregnancy, reviews current evidence, compares the guideline recommendations with respect to each key component, and provides our preferred diagnostic pathway. It provides the guidelines and available data needed for informed decision making to diagnose pulmonary embolism in pregnancy.

Evaluation of CTPA interpreted as limited in pregnant patients suspected for pulmonary embolism

PURPOSE

The purpose of this study is to determine the rates of CT pulmonary angiography (CTPA) interpreted as limited and severely limited in pregnant patients suspected for pulmonary embolism (PE), and to evaluate factors that influence these rates.

METHODS

This is a retrospective study with CTPA for evaluation of PE in pregnancy across a large health system from 2006 to 2017. CTPA was classified as limited from the radiology report with a subset of those studies classified as severely limited. Bivariate and multivariate analysis was performed for limited and severely limited rates with maternal age and patient size as a continuous variable and race, trimester, patient location study priority status, and result of chest radiograph before CTPA as categorical variables.

RESULTS

874 patients with 33% of studies limited and 4% of studies severely limited. Multivariate logistic regression of CTPA studies revealed decreasing patient age (OR 0.967, p = 0.0129) and increasing patient size (OR 1.013, p < 0.0001). Studies performed in the second trimester (OR 1.869, p = 0.0242) and third trimester (OR 2.314, p = 0.0021) were more likely to be reported as limited (each p < 0.05). Increasing patient size (OR 1.017, p = 0.0046) was the only significant predictor of severely limited versus non-severely limited studies.

CONCLUSION

CTPA interpreted as limited in pregnancy are common and may be associated with younger age, larger patient size, and second and third trimesters. However, severely limited interpretations are much less common, with patient size the only significant predictor.

Using normalisation process theory to understand workflow implications of decision support implementation across diverse primary care settings

BACKGROUND

Effective implementation of technologies into clinical workflow is hampered by lack of integration into daily activities. Normalisation process theory (NPT) can be used to describe the kinds of 'work' necessary to implement and embed complex new practices. We determined the suitability of NPT to assess the facilitators, barriers and 'work' of implementation of two clinical decision support (CDS) tools across diverse care settings.

METHODS

We conducted baseline and 6-month follow-up quantitative surveys of clinic leadership at two academic institutions' primary care clinics randomised to the intervention arm of a larger study. The survey was adapted from the NPT toolkit, analysing four implementation domains: sense-making, participation, action, monitoring. Domains were summarised among completed responses (n=60) and examined by role, institution, and time.

RESULTS

The median score for each NPT domain was the same across roles and institutions at baseline, and decreased at 6 months. At 6 months, clinic managers' participation domain (p=0.003), and all domains for medical directors (p<0.003) declined. At 6 months, the action domain decreased among Utah respondents (p=0.03), and all domains decreased among Wisconsin respondents (p≤0.008).

CONCLUSIONS

This study employed NPT to longitudinally assess the implementation barriers of new CDS. The consistency of results across participant roles suggests similarities in the work each role took on during implementation. The decline in engagement over time suggests the need for more frequent contact to maintain momentum. Using NPT to evaluate this implementation provides insight into domains which can be addressed with participants to improve success of new electronic health record technologies.

TRIAL REGISTRATION NUMBER

NCT02534987.

Abstract 936: Entropyx of Cardiac Rhythm Uniquely Predicts Mortality in a Multi-site Polysomnography (psg) Study of Ambulatory Asymptomatic Community Adults With Heart Failure

Introduction: Heart failure (HF) is associated with high rates of mortality and hospital readmission. Current strategies for risk stratification are limited. Recently, we introduced EntropyX, a novel measure of non-linear patterns underlying physiological variability using newer concepts of entropy estimation and machine learning. EntropyX of cardiac repolarization (EntropyX QT ) enhanced the predictive value of all established risk factors in a multicenter study of HF patients (PMID: 27044982). Herein, we test the hypothesis that EntropyX of ventricular activation (EntropyX RR ) during sleep enhances the performance of established mortality risk factors in asymptomatic community adults with HF. We interpret our results in context of fundamental mechanistic studies in animal models and modern theories of systems biology.

Methods & Results: We followed 96 NYHA class I HF adults in sinus rhythm for 6.5±3.0 years (1994-2011; SHHS NCT#00005275). Baseline exposures included demographics, history, medications, labs, PSG metrics [e.g., sleep disordered breathing (SDB)], ECG analyses [e.g., heart rate (HRV), QT variability (QTV)], and EntropyX RR . The cohort had mean age of 70±10 years, 49% women, 11% African Americans, and 46 deaths (48%; N=35 from cardiovascular events) over 4.6±2.6 years. After adjusting for exposures, the adjusted hazard ratios (4th to 1st quartile) for mortality for EntropyX RR was 2.3 (95% CI, 1.1-4.6) and age was 4.5 (1.9-10.3), consistent with machine learning-based classification and regression tree analysis. Addition of EntropyX RR to a multivariate model (comprised of age, diabetes, myocardial infarction, SDB) improved continuous net reclassification by 43% (37-48).

Conclusions: EntropyX RR during sleep predicts mortality over follow-up of asymptomatic community adults with HF, independent of conventional risk factors and linear/deterministic measures (e.g., HRV, QTV, SDB). Unlike simpler concepts of RR variability, EntropyX RR is a fundamentally different measure, reflecting the complexity of multi-directional physiological network properties that regulate homeostatic function under changing conditions. This new paradigm complements conventional measures of risk and has potential for broad application.

Live Usability Testing of Two Complex Clinical Decision Support Tools: Observational Study

BACKGROUND

Potential of the electronic health records (EHR) and clinical decision support (CDS) systems to improve the practice of medicine has been tempered by poor design and the resulting burden they place on providers. CDS is rarely tested in the real clinical environment. As a result, many tools are hard to use, placing strain on providers and resulting in low adoption rates. The existing CDS usability literature relies primarily on expert opinion and provider feedback via survey. This is the first study to evaluate CDS usability and the provider-computer-patient interaction with complex CDS in the real clinical environment.

OBJECTIVE

This study aimed to further understand the barriers and facilitators of meaningful CDS usage within a real clinical context.

METHODS

This qualitative observational study was conducted with 3 primary care providers during 6 patient care sessions. In patients with the chief complaint of sore throat, a CDS tool built with the Centor Score was used to stratify the risk of group A Streptococcus pharyngitis. In patients with a chief complaint of cough or upper respiratory tract infection, a CDS tool built with the Heckerling Rule was used to stratify the risk of pneumonia. During usability testing, all human-computer interactions, including audio and continuous screen capture, were recorded using the Camtasia software. Participants' comments and interactions with the tool during clinical sessions and participant comments during a postsession brief interview were placed into coding categories and analyzed for generalizable themes.

RESULTS

In the 6 encounters observed, primary care providers toggled between addressing either the computer or the patient during the visit. Minimal time was spent listening to the patient without engaging the EHR. Participants mostly used the CDS tool with the patient, asking questions to populate the calculator and discussing the results of the risk assessment; they reported the ability to do this as the major benefit of the tool. All providers were interrupted during their use of the CDS tool by the need to refer to other sections of the chart. In half of the visits, patients' clinical symptoms challenged the applicability of the tool to calculate the risk of bacterial infection. Primary care providers rarely used the incorporated incentives for CDS usage, including progress notes and patient instructions.

CONCLUSIONS

Live usability testing of these CDS tools generated insights about their role in the patient-provider interaction. CDS may contribute to the interaction by being simultaneously viewed by the provider and patient. CDS can improve usability and lessen the strain it places on providers by being short, flexible, and customizable to unique provider workflow. A useful component of CDS is being as widely applicable as possible and ensuring that its functions represent the fastest way to perform a particular task.

Improving Provider Adoption With Adaptive Clinical Decision Support Surveillance: An Observational Study

Background

Successful clinical decision support (CDS) tools can help use evidence-based medicine to effectively improve patient outcomes. However, the impact of these tools has been limited by low provider adoption due to overtriggering, leading to alert fatigue. We developed a tracking mechanism for monitoring trigger (percent of total visits for which the tool triggers) and adoption (percent of completed tools) rates of a complex CDS tool based on the Wells criteria for pulmonary embolism (PE).

Objective

We aimed to monitor and evaluate the adoption and trigger rates of the tool and assess whether ongoing tool modifications would improve adoption rates.

Methods

As part of a larger clinical trial, a CDS tool was developed using the Wells criteria to calculate pretest probability for PE at 2 tertiary centers’ emergency departments (EDs). The tool had multiple triggers: any order for D-dimer, computed tomography (CT) of the chest with intravenous contrast, CT pulmonary angiography (CTPA), ventilation-perfusion scan, or lower extremity Doppler ultrasound. A tracking dashboard was developed using Tableau to monitor real-time trigger and adoption rates. Based on initial low provider adoption rates of the tool, we conducted small focus groups with key ED providers to elicit barriers to tool use. We identified overtriggering of the tool for non-PE-related evaluations and inability to order CT testing for intermediate-risk patients. Thus, the tool was modified to allow CT testing for the intermediate-risk group and not to trigger for CT chest with intravenous contrast orders. A dialogue box, “Are you considering PE for this patient?” was added before the tool triggered to account for CTPAs ordered for aortic dissection evaluation.

Results

In the ED of tertiary center 1, 95,295 patients visited during the academic year. The tool triggered for an average of 509 patients per month (average trigger rate 2036/30,234, 6.73%) before the modifications, reducing to 423 patients per month (average trigger rate 1629/31,361, 5.22%). In the ED of tertiary center 2, 88,956 patients visited during the academic year, with the tool triggering for about 473 patients per month (average trigger rate 1892/29,706, 6.37%) before the modifications and for about 400 per month (average trigger rate 1534/30,006, 5.12%) afterward. The modifications resulted in a significant 4.5- and 3-fold increase in provider adoption rates in tertiary centers 1 and 2, respectively. The modifications increased the average monthly adoption rate from 23.20/360 (6.5%) tools to 81.60/280.20 (29.3%) tools and 46.60/318.80 (14.7%) tools to 111.20/263.40 (42.6%) tools in centers 1 and 2, respectively.

Conclusions

Close postimplementation monitoring of CDS tools may help improve provider adoption. Adaptive modifications based on user feedback may increase targeted CDS with lower trigger rates, reducing alert fatigue and increasing provider adoption. Iterative improvements and a postimplementation monitoring dashboard can significantly improve adoption rates.

Adaptive design of a clinical decision support tool: What the impact on utilization rates means for future CDS research

OBJECTIVE

We employed an agile, user-centered approach to the design of a clinical decision support tool in our prior integrated clinical prediction rule study, which achieved high adoption rates. To understand if applying this user-centered process to adapt clinical decision support tools is effective in improving the use of clinical prediction rules, we examined utilization rates of a clinical decision support tool adapted from the original integrated clinical prediction rule study tool to determine if applying this user-centered process to design yields enhanced utilization rates similar to the integrated clinical prediction rule study.

MATERIALS & METHODS: We conducted pre-deployment usability testing and semi-structured group interviews at 6 months post-deployment with 75 providers at 14 intervention clinics across the two sites to collect user feedback. Qualitative data analysis is bifurcated into immediate and delayed stages; we reported on immediate-stage findings from real-time field notes used to generate a set of rapid, pragmatic recommendations for iterative refinement. Monthly utilization rates were calculated and examined over 12 months.

RESULTS

We hypothesized a well-validated, user-centered clinical decision support tool would lead to relatively high adoption rates. Then 6 months post-deployment, integrated clinical prediction rule study tool utilization rates were substantially lower than anticipated based on the original integrated clinical prediction rule study trial (68%) at 17% (Health System A) and 5% (Health System B). User feedback at 6 months resulted in recommendations for tool refinement, which were incorporated when possible into tool design; however, utilization rates at 12 months post-deployment remained low at 14% and 4% respectively.

DISCUSSION

Although valuable, findings demonstrate the limitations of a user-centered approach given the complexity of clinical decision support.

CONCLUSION

Strategies for addressing persistent external factors impacting clinical decision support adoption should be considered in addition to the user-centered design and implementation of clinical decision support.

A Computerized Method for Measuring Computed Tomography Pulmonary Angiography Yield in the Emergency Department: Validation Study

Background

Use of computed tomography pulmonary angiography (CTPA) in the assessment of pulmonary embolism (PE) has markedly increased over the past two decades. While this technology has improved the accuracy of radiological testing for PE, CTPA also carries the risk of substantial iatrogenic harm. Each CTPA carries a 14% risk of contrast-induced nephropathy and a lifetime malignancy risk that can be as high as 2.76%. The appropriate use of CTPA can be estimated by monitoring the CTPA yield, the percentage of tests positive for PE. This is the first study to propose and validate a computerized method for measuring the CTPA yield in the emergency department (ED).

Objective

The objective of our study was to assess the validity of a novel computerized method of calculating the CTPA yield in the ED.

Methods

The electronic health record databases at two tertiary care academic hospitals were queried for CTPA orders completed in the ED over 1-month periods. These visits were linked with an inpatient admission with a discharge diagnosis of PE based on the International Classification of Diseases codes. The computerized the CTPA yield was calculated as the number of CTPA orders with an associated inpatient discharge diagnosis of PE divided by the total number of orders for completed CTPA. This computerized method was then validated by 2 independent reviewers performing a manual chart review, which included reading the free-text radiology reports for each CTPA.

Results

A total of 349 CTPA orders were completed during the 1-month periods at the two institutions. Of them, acute PE was diagnosed on CTPA in 28 studies, with a CTPA yield of 7.7%. The computerized method correctly identified 27 of 28 scans positive for PE. The one discordant scan was tied to a patient who was discharged directly from the ED and, as a result, never received an inpatient discharge diagnosis.

Conclusions

This is the first successful validation study of a computerized method for calculating the CTPA yield in the ED. This method for data extraction allows for an accurate determination of the CTPA yield and is more efficient than manual chart review. With this ability, health care systems can monitor the appropriate use of CTPA and the effect of interventions to reduce overuse and decrease preventable iatrogenic harm.

First Results from the DEAP-3600 Dark Matter Search with Argon at SNOLAB

This Letter reports the first results of a direct dark matter search with the DEAP-3600 single-phase liquid argon (LAr) detector. The experiment was performed 2 km underground at SNOLAB (Sudbury, Canada) utilizing a large target mass, with the LAr target contained in a spherical acrylic vessel of 3600 kg capacity. The LAr is viewed by an array of PMTs, which would register scintillation light produced by rare nuclear recoil signals induced by dark matter particle scattering. An analysis of 4.44 live days (fiducial exposure of 9.87 ton day) of data taken during the initial filling phase demonstrates the best electronic recoil rejection using pulse-shape discrimination in argon, with leakage <1.2×10^{-7} (90% C.L.) between 15 and 31  keV_{ee}. No candidate signal events are observed, which results in the leading limit on weakly interacting massive particle (WIMP)-nucleon spin-independent cross section on argon, <1.2×10^{-44}  cm^{2} for a 100  GeV/c^{2} WIMP mass (90% C.L.).

Do clinicians want recommendations? A multicenter study comparing evidence summaries with and without GRADE recommendations

OBJECTIVES

Evidence-based clinical practice guidelines provide recommendations to assist clinicians in decision-making and to reduce the gap between best current research evidence and clinical practice. However, some argue that providing preappraised evidence summaries alone, rather than recommendations, is more appropriate. The objective of the study is to evaluate clinicians' preferences, and understanding of the evidence and intended course of action in response to evidence summaries with and without recommendations.

STUDY DESIGN SETTING

We included practicing clinicians attending educational sessions across 10 countries. Clinicians were randomized to receive relevant clinical scenarios supported by research evidence of low or very low certainty and accompanied by either strong or weak recommendations developed with the GRADE system. Within each group, participants were further randomized to receive the recommendation plus the corresponding evidence summary or the evidence summary alone. We evaluated participants' preferences and understanding for the presentation strategy, as well as their intended course of action.

RESULTS

One hundred eighty-nine of 219 (86%) and 201 of 248 (81%) participants preferred having recommendations accompanying evidence summaries for both strong and weak recommendations, respectively. Across all scenarios, less than half of participants correctly interpreted information provided in the evidences summaries (e.g., estimates of effect, certainty in the research evidence). The presence of a recommendation resulted in a more appropriate intended course of action for two scenarios involving strong recommendations.

CONCLUSION

Evidence summaries alone are not enough to impact clinicians' course of action. Clinicians clearly prefer having recommendations accompanying evidence summaries in the context of low or very low certainty of evidence (Trial registration NCT02006017).

Mortality and Hospitalizations in Intensive Dialysis: A Systematic Review and Meta-Analysis

Background:

Survival and hospitalization are critically important outcomes considered when choosing between intensive hemodialysis (HD), conventional HD, and peritoneal dialysis (PD). However, the comparative effectiveness of these modalities is unclear.

Objective:

We had the following aims: (1) to compare the association of mortality and hospitalization in patients undergoing intensive HD, compared with conventional HD or PD and (2) to appraise the methodological quality of the supporting evidence.

Data Sources:

MEDLINE, Embase, ISI Web of Science, CENTRAL, and nephrology conference abstracts.

Study Eligibility, Participants, and Interventions:

We included cohort studies with comparator arm, and randomized controlled trials (RCTs) with >50% of adult patients (≥18 years) comparing any form of intensive HD (>4 sessions/wk or >5.5 h/session) with any form of chronic dialysis (PD, HD ≤4 sessions/wk or ≤5.5 h/session), that reported at least 1 predefined outcome (mortality or hospitalization).

Methods:

We used the GRADE approach to systematic reviews and quality appraisal. Two reviewers screened citations and full-text articles, and extracted study-level data independently, with discrepancies resolved by consensus. We pooled effect estimates of randomized and observational studies separately using generic inverse variance with random effects models, and used fixed-effects models when only 2 studies were available for pooling. Predefined subgroups for the intensive HD cohorts were classified by nocturnal versus short daily HD and home versus in-center HD.

Results:

Twenty-three studies with a total of 70 506 patients were included. Of the observational studies, compared with PD, intensive HD had a significantly lower mortality risk (hazard ratio [HR]: 0.67; 95% confidence interval [CI]: 0.53-0.84; I² = 91%). Compared with conventional HD, home nocturnal (HR: 0.46; 95% CI: 0.38-0.55; I² = 0%), in-center nocturnal (HR: 0.73; 95% CI: 0.60-0.90; I² = 57%) and home short daily (HR: 0.54; 95% CI: 0.31-0.95; I² = 82%) intensive regimens had lower mortality. Of the 2 RCTs assessing mortality, in-center short daily HD had lower mortality (HR: 0.54; 95% CI: 0.31-0.93), while home nocturnal HD had higher mortality (HR: 3.88; 95% CI: 1.27-11.79) in long-term observational follow-up. Hospitalization days per patient-year (mean difference: –1.98; 95% CI: –2.37 to −1.59; I² = 6%) were lower in nocturnal compared with conventional HD. Quality of evidence was similarly low or very low in RCTs (due to imprecision) and observational studies (due to residual confounding and selection bias).

Limitations:

The overall quality of evidence was low or very low for critical outcomes. Outcomes such as quality of life, transplantation, and vascular access outcomes were not included in our review.

Conclusions:

Intensive HD regimens may be associated with reduced mortality and hospitalization compared with conventional HD or PD. As the quality of supporting evidence is low, patients who place a high value on survival must be adequately advised and counseled of risks and benefits when choosing intensive dialysis. Practice guidelines that promote shared decision-making are likely to be helpful.

Potentially Avoidable Readmissions in United States Hemodialysis Patients

Introduction

Patients with end-stage kidney disease have a high risk of 30-day readmission to hospital. These readmissions are financially costly to health care systems and are associated with poor health-related quality of life. The objective of this study was to describe and analyze the frequency, causes, and predictors of 30-day potentially avoidable readmission to hospital in patients on hemodialysis.

Methods

We conducted a retrospective cohort study using the US Renal Data System data from January 1, 2008, to December 31, 2008. A total of 107,940 prevalent United States hemodialysis patients with 248,680 index hospital discharges were assessed for the main outcome of 30-day potentially avoidable readmission, as identified by a computerized algorithm.

Results

Of 83,209 30-day readmissions, 59,045 (70.1%) resulted in a 30-day potentially avoidable readmission. The geographic distribution of 30-day potentially avoidable readmission in the United States varied by state. Characteristics associated with 30-day potentially avoidable readmission included the following: younger age, shorter time on hemodialysis, at least 3 or more hospitalizations in preceding 12 months, black race, unemployed status, treatment at a for-profit facility, longer length of index hospital stay, and index hospitalizations that involved a surgical procedure. The 5-, 15-, and 30-day potentially avoidable readmission cumulative incidences were 6.0%, 15.1%, and 25.8%, respectively.

Conclusion

Patients with end-stage kidney disease on maintenance hemodialysis are at high risk for 30-day readmission to hospital, with nearly three-quarters (70.1%) of all 30-day readmissions being potentially avoidable. Research is warranted to develop cost-effective and transferrable interventions that improve care transitions from hospital to outpatient hemodialysis facility and reduce readmission risk for this vulnerable population.

Discrimination and Calibration of Clinical Prediction Models: Users' Guides to the Medical Literature

Accurate information regarding prognosis is fundamental to optimal clinical care. The best approach to assess patient prognosis relies on prediction models that simultaneously consider a number of prognostic factors and provide an estimate of patients' absolute risk of an event. Such prediction models should be characterized by adequately discriminating between patients who will have an event and those who will not and by adequate calibration ensuring accurate prediction of absolute risk. This Users' Guide will help clinicians understand the available metrics for assessing discrimination, calibration, and the relative performance of different prediction models. This article complements existing Users' Guides that address the development and validation of prediction models. Together, these guides will help clinicians to make optimal use of existing prediction models.

"Think aloud" and "Near live" usability testing of two complex clinical decision support tools

OBJECTIVES

Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. "Think Aloud" and "Near Live" usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to "Near Live" testing after completing "Think Aloud".

METHODS

This was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During "Think Aloud" testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During "Near Live" testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods.

RESULTS

"Think Aloud" and "Near Live" usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during "Near Live" testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during "Near-Live" testing because they would not be all inclusive for the visit.

CONCLUSIONS

These complementary types of usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. The additional feedback from "Near Live" testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.

Design and implementation of electronic health record integrated clinical prediction rules (iCPR): a randomized trial in diverse primary care settings

Background

Clinical prediction rules (CPRs) represent a method of determining individual patient risk to help providers make more accurate decisions at the point of care. Well-validated CPRs are underutilized but may decrease antibiotic overuse for acute respiratory infections. The integrated clinical prediction rules (iCPR) study builds on a previous single clinic study to integrate two CPRs into the electronic health record and assess their impact on practice. This article discusses study design and implementation of a multicenter cluster randomized control trial of the iCPR clinical decision support system, including the tool adaptation, usability testing, staff training, and implementation study to disseminate iCPR at multiple clinical sites across two health care systems.

Methods

The iCPR tool is based on two well-validated CPRs, one for strep pharyngitis and one for pneumonia. The iCPR tool uses the reason for visit to trigger a risk calculator. Provider completion of the risk calculator provides a risk score, which is linked to an order set. Order sets guide evidence-based care and include progress note documentation, tests, prescription medications, and patient instructions. The iCPR tool was refined based on interviews with providers, medical assistants, and clinic managers, and two rounds of usability testing. “Near live” usability testing with simulated patients was used to ensure that iCPR fit into providers’ clinical workflows. Thirty-three Family Medicine and General Internal Medicine primary care clinics were recruited at two institutions. Clinics were randomized to academic detailing about strep pharyngitis and pneumonia diagnosis and treatment (control) or academic detailing plus use of the iCPR tool (intervention). The primary outcome is the difference in antibiotic prescribing rates between the intervention and control groups with secondary outcomes of difference in rapid strep and chest x-ray ordering. Use of the components of the iCPR will also be assessed.

Discussion

The iCPR study uses a strong user-centered design and builds on the previous initial study, to assess whether CPRs integrated in the electronic health record can change provider behavior and improve evidence-based care in a broad range of primary care clinics.

Trial registration

Clinicaltrials.gov (NCT02534987)

Cardiorespiratory Fitness in Internal Medicine Residents: Are Future Physicians Becoming Deconditioned?

BACKGROUND

Previous studies have shown a falloff in physicians' physical activity from medical school to residency. Poor fitness may result in stress, increase resident burnout, and contribute to mortality from cardiovascular disease and other causes. Physicians with poor exercise habits are also less likely to counsel patients about exercise. Prior studies have reported resident physical activity but not cardiorespiratory fitness age.

OBJECTIVE

The study was conducted in 2 residency programs (3 hospitals) to assess internal medicine residents' exercise habits as well as their cardiorespiratory fitness age.

METHODS

Data regarding physical fitness levels and exercise habits were collected in an anonymous cross-sectional survey. Cardiopulmonary fitness age was determined using fitness calculator based on the Nord-Trøndelag Health Study (HUNT).

RESULTS

Of 199 eligible physicians, 125 (63%) responded to the survey. Of respondents, 11 (9%) reported never having exercised prior to residency and 45 (36%) reported not exercising during residency (P < .001). In addition, 42 (34%) reported exercising every day prior to residency, while only 5 (4%) reported exercising daily during residency (P < .001), with 99 (79%) participants indicating residency obligations as their main barrier to exercise. We found residents' calculated mean fitness age to be 5.6 years higher than their mean chronological age (P < .001).

CONCLUSIONS

Internal medicine residents reported significant decreases in physical activity and fitness. Residents attributed time constraints due to training as a key barrier to physical activity.

Other Ways of Knowing

BACKGROUND

Patients with advanced-stage chronic obstructive pulmonary disease (COPD) may suffer severe respiratory exacerbations and need to decide between accepting life-sustaining treatments versus foregoing these treatments (choosing comfort care only). We designed the InformedTogether decision aid to inform this decision and describe results of a pilot study to assess usability focusing on participants' trust in the content of the decision aid, acceptability, recommendations for improvement, and emotional reactions to this emotionally laden decision.

METHODS

Study participants ( N = 26) comprising clinicians, patients, and surrogates viewed the decision aid, completed usability tasks, and participated in interviews and focus groups assessing comprehension, trust, perception of bias, and perceived acceptability of InformedTogether. Mixed methods were used to analyze results.

RESULTS

Almost all participants understood the gist (general meaning) of InformedTogether. However, many lower literacy participants had difficulty answering the more detailed questions related to comprehension, especially when interpreting icon arrays, and many were not aware that they had misunderstood the information. Qualitative analysis showed a range of emotional reactions to the information. Participants with low verbatim comprehension frequently referenced lived experiences when answering knowledge questions, which we termed "alternative knowledge."

CONCLUSIONS

We found a range of emotional reactions to the information and frequent use of alternative knowledge frameworks for deriving meaning from the data. These observations led to insights into the impact of lived experiences on the uptake of biomedical information presented in decision aids. Communicating prognostic information could potentially be improved by eliciting alternative knowledge as a starting point to build communication, in particular for low literacy patients. Decision aids designed to facilitate shared decision making should elicit this knowledge and help clinicians tailor information accordingly.

Avoiding alert fatigue in pulmonary embolism decision support: a new method to examine 'trigger rates'

A clinical decision support system (CDSS) is integrated into the electronic health record (EHR) and allows physicians to easily use a clinical decision support (CDS) tool. However, often CDSSs are integrated into the EHR with poor adoption rates. One reason for this is secondary to 'trigger fatigue'. Therefore, we developed a new and innovative usability process named 'sensitivity and specificity trigger analysis' (SSTA) as part of our larger project around a pulmonary embolism decision support tool. SSTA will enable programmers to examine optimal trigger rates prior to the integration of a CDS tool into the EHR, by using a formal method of analysis. We performed a retrospective chart review. The outcome of interest was physician ordering of a CT angiography (CTA). Phrases that signify common symptoms associated with pulmonary embolism were assessed as possible triggers for the CDSS tool. We then analysed each trigger's ability to predict physician ordering of a CTA. We found that the most sensitive way to trigger the Pulmonary Embolism CDS tool while still maintaining a high specificity was by combining 1 or more pertinent symptoms with 1 or more elements of the Wells criteria. This study explored a unique methodology, SSTA, used to limit inaccurate triggering of a CDS tool prior to integration into the EHR. This methodology can be applied to other studies aiming to decrease triggering rates and increase adoption rates of previously validated CDSS tools.

An Electronic Adherence Measurement Intervention to Reduce Clinical Inertia in the Treatment of Uncontrolled Hypertension: The MATCH Cluster Randomized Clinical Trial

BACKGROUND

To appropriately manage uncontrolled hypertension, clinicians must decide whether blood pressure (BP) is above goal due to a need for additional medication or to medication nonadherence. Yet, clinicians are poor judges of adherence, and uncertainty about adherence may promote inertia with respect to medication modification.

OBJECTIVE

We aimed to determine the effect of sharing electronically-measured adherence data with clinicians on the management of uncontrolled hypertension.

DESIGN

This was a cluster randomized trial.

PARTICIPANTS

Twenty-four primary care providers (12 intervention, 12 usual care; cluster units) and 100 patients with uncontrolled hypertension (65 intervention, 35 usual care) were included in the study.

INTERVENTIONS

At one visit per patient, clinicians in the intervention group received a report summarizing electronically measured adherence to the BP regimen and recommended clinical actions. Clinicians in the control group did not receive a report.

MAIN MEASURES

The primary outcome was the proportion of visits with appropriate clinical management (i.e., treatment intensification among adherent patients and adherence counseling among nonadherent patients). Secondary outcomes included patient-rated quality of care and communication during the visit.

KEY RESULTS

The proportion of visits with appropriate clinical management was higher in the intervention group than the control group (45 out of 65; 69 %) versus (12 out of 35; 34 %; p = 0.001). A higher proportion of adherent patients in the intervention group had their regimen intensified (p = 0.01), and a higher proportion of nonadherent patients in the intervention group received adherence counseling (p = 0.005). Patients in the intervention group were more likely to give their clinician high ratings on quality of care (p = 0.05), and on measures of patient-centered (p = 0.001) and collaborative communication (p = 0.02).

CONCLUSIONS

Providing clinicians with electronically-measured antihypertensive adherence reports reduces inertia in the management of uncontrolled hypertension.

TRIAL REGISTRATION

NCT01257347 ; http://clinicaltrials.gov/show/ NCT01257347.

Perioperative Management of Patients on Oral Anticoagulants: A Decision Analysis

Background . To better inform clinicians on the optimal management of patients on oral anticoagulation who need to undergo surgery or invasive procedures, the authors performed a decision analysis examining whether a perioperative aggressive or minimalist strategy results in greater quality-adjusted survival. Methods . A decision analysis model was created comparing withholding warfarin (minimalist strategy) to withholding warfarin and administering treatment-dose subcutaneous low-molecular-weight heparin (LMWH) or intravenous heparin perioperatively (aggressive strategy). The base-case analysis examined a hypothetical 60-year-old hypertensive individual with mechanical aortic valve replacement undergoing major abdominal surgery. A probabilistic sensitivity analysis was performed using a Monte Carlo simulation with quality-adjusted life expectancy (QALE) as the outcome. Secondary analyses examined patients with a mechanical mitral valve and atrial fibrillation. Sensitivity analyses were performed for each variable. Results . Under the base-case scenario, the minimalist strategy was preferred for 78% of trials in the Monte Carlo simulation, with a mean benefit of 0.003 years (95% confidence interval, -0.005 years to 0.011 years). Sensitivity analyses based on point estimates indicate that the aggressive strategy is preferred when the annual stroke rate is >5.6% or the increase in postoperative major bleeding induced by heparin is <2.0%; however, the benefit is small over the range of plausible values. Conclusions . For most patients with a mechanical aortic valve or atrial fibrillation undergoing major surgery, a minimalist strategy of simply withholding oral anticoagulation provides similar QALE as an aggressive strategy of administering perioperative subcutaneous LMWH or intravenous heparin. The aggressive therapy provides greater QALE for patients at higher risk of stroke (e.g., mechanical mitral valves), although the benefit is small.

External validation of the IMPROVE Bleeding Risk Assessment Model in medical patients

The IMPROVE Bleed Risk Assessment Model (RAM) remains the only bleed RAM in hospitalised medical patients using 11 clinical and laboratory factors. The aim of our study was to externally validate the IMPROVE Bleed RAM. A retrospective chart review was conducted between October 1, 2012 and July 31, 2014. We applied the point scoring system to compute risk scores for each patient in the validation sample. We then dichotomised the patients into those with a score <7 (low risk) vs ≥ 7 (high risk), as outlined in the original study, and compared the rates of any bleed, non-major bleed, and major bleed. Among the 12,082 subjects, there was an overall 2.6 % rate of any bleed within 14 days of admission. There was a 2.12 % rate of any bleed in those patients with a score of < 7 and a 4.68 % rate in those with a score ≥ 7 [Odds Ratio (OR) 2.3 (95 % CI=1.8-2.9), p<0.0001]. MB rates were 1.5 % in the patients with a score of < 7 and 3.2 % in the patients with a score of ≥ 7, [OR 2.2 (95 % CI=1.6-2.9), p<0.0001]. The ROC curve was 0.63 for the validation sample. This study represents the largest externally validated Bleed RAM in a hospitalised medically ill patient population. A cut-off point score of 7 or above was able to identify a high-risk patient group for MB and any bleed. The IMPROVE Bleed RAM has the potential to allow for more tailored approaches to thromboprophylaxis in medically ill hospitalised patients.

Putting Meaning into Meaningful Use: A Roadmap to Successful Integration of Evidence at the Point of Care

Pressures to contain health care costs, personalize patient care, use big data, and to enhance health care quality have highlighted the need for integration of evidence at the point of care. The application of evidence-based medicine (EBM) has great promise in the era of electronic health records (EHRs) and health technology. The most successful integration of evidence into EHRs has been complex decision tools that trigger at a critical point of the clinical visit and include patient specific recommendations. The objective of this viewpoint paper is to investigate why the incorporation of complex CDS tools into the EMR is equally complex and continues to challenge health service researchers and implementation scientists. Poor adoption and sustainability of EBM guidelines and CDS tools at the point of care have persisted and continue to document low rates of usage. The barriers cited by physicians include efficiency, perception of usefulness, information content, user interface, and over-triggering. Building on the traditional EHR implementation frameworks, we review keys strategies for successful CDSs: (1) the quality of the evidence, (2) the potential to reduce unnecessary care, (3) ease of integrating evidence at the point of care, (4) the evidence’s consistency with clinician perceptions and preferences, (5) incorporating bundled sets or automated documentation, and (6) shared decision making tools. As EHRs become commonplace and insurers demand higher quality and evidence-based care, better methods for integrating evidence into everyday care are warranted. We have outlined basic criteria that should be considered before attempting to integrate evidenced-based decision support tools into the EHR.

Formative assessment and design of a complex clinical decision support tool for pulmonary embolism

Electronic health record (EHR)-based clinical decision support (CDS) tools are rolled out with the urgency to meet federal requirements without time for usability testing and refinement of the user interface. As part of a larger project to design, develop and integrate a pulmonary embolism CDS tool for emergency physicians, we conducted a formative assessment to determine providers' level of interest and input on designs and content. This was a study to conduct a formative assessment of emergency medicine (EM) physicians that included focus groups and key informant interviews. The focus of this study was twofold, to determine the general attitude towards CDS tool integration and the ideal integration point into the clinical workflow. To accomplish this, we first approached EM physicians in a focus group, then, during key informant interviews, we presented workflow designs and gave a scenario to help the providers visualise how the CDS tool works. Participants were asked questions regarding the trigger location, trigger words, integration into their workflow, perceived utility and heuristic of the tool. Results from the participants' survey responses to trigger location, perceived utility and efficiency, indicated that the providers felt the tool would be more of a hindrance than an aid. However, some providers commented that they had not had exposure to CDS tools but had used online calculators, and thought the tools would be helpful at the point-of-care if integrated into the EHR. Furthermore, there was a preference for an order entry wireframe. This study highlights several factors to consider when designing CDS tools: (1) formative assessment of EHR functionality and clinical environment workflow, (2) focus groups and key informative interviews to incorporate providers' perceptions of CDS and workflow integration and/or (3) the demonstration of proposed workflows through wireframes to help providers visualise design concepts.

Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned

Background

As the electronic health record (EHR) becomes the preferred documentation tool across medical practices, health care organizations are pushing for clinical decision support systems (CDSS) to help bring clinical decision support (CDS) tools to the forefront of patient-physician interactions. A CDSS is integrated into the EHR and allows physicians to easily utilize CDS tools. However, often CDSS are integrated into the EHR without an initial phase of usability testing, resulting in poor adoption rates. Usability testing is important because it evaluates a CDSS by testing it on actual users. This paper outlines the usability phase of a study, which will test the impact of integration of the Wells CDSS for pulmonary embolism (PE) diagnosis into a large urban emergency department, where workflow is often chaotic and high stakes decisions are frequently made. We hypothesize that conducting usability testing prior to integration of the Wells score into an emergency room EHR will result in increased adoption rates by physicians.

Objective

The objective of the study was to conduct usability testing for the integration of the Wells clinical prediction rule into a tertiary care center’s emergency department EHR.

Methods

We conducted usability testing of a CDS tool in the emergency department EHR. The CDS tool consisted of the Wells rule for PE in the form of a calculator and was triggered off computed tomography (CT) orders or patients’ chief complaint. The study was conducted at a tertiary hospital in Queens, New York. There were seven residents that were recruited and participated in two phases of usability testing. The usability testing employed a “think aloud” method and “near-live” clinical simulation, where care providers interacted with standardized patients enacting a clinical scenario. Both phases were audiotaped, video-taped, and had screen-capture software activated for onscreen recordings.

Results

Phase I: Data from the “think-aloud” phase of the study showed an overall positive outlook on the Wells tool in assessing a patient for a PE diagnosis. Subjects described the tool as “well-organized” and “better than clinical judgment”. Changes were made to improve tool placement into the EHR to make it optimal for decision-making, auto-populating boxes, and minimizing click fatigue. Phase II: After incorporating the changes noted in Phase 1, the participants noted tool improvements. There was less toggling between screens, they had all the clinical information required to complete the tool, and were able to complete the patient visit efficiently. However, an optimal location for triggering the tool remained controversial.

Conclusions

This study successfully combined “think-aloud” protocol analysis with “near-live” clinical simulations in a usability evaluation of a CDS tool that will be implemented into the emergency room environment. Both methods proved useful in the assessment of the CDS tool and allowed us to refine tool usability and workflow.

Healthcare provider perceptions of clinical prediction rules

OBJECTIVES

To examine internal medicine and emergency medicine healthcare provider perceptions of usefulness of specific clinical prediction rules.

SETTING

The study took place in two academic medical centres. A web-based survey was distributed and completed by participants between 1 January and 31 May 2013.

PARTICIPANTS

Medical doctors, doctors of osteopathy or nurse practitioners employed in the internal medicine or emergency medicine departments at either institution.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome was to identify the clinical prediction rules perceived as most useful by healthcare providers specialising in internal medicine and emergency medicine. Secondary outcomes included comparing usefulness scores of specific clinical prediction rules based on provider specialty, and evaluating associations between usefulness scores and perceived characteristics of these clinical prediction rules.

RESULTS

Of the 401 healthcare providers asked to participate, a total of 263 (66%), completed the survey. The CHADS2 score was chosen by most internal medicine providers (72%), and Pulmonary Embolism Rule-Out Criteria (PERC) score by most emergency medicine providers (45%), as one of the top three most useful from a list of 24 clinical prediction rules. Emergency medicine providers rated their top three significantly more positively, compared with internal medicine providers, as having a better fit into their workflow (p=0.004), helping more with decision-making (p=0.037), better fitting into their thought process when diagnosing patients (p=0.001) and overall, on a 10-point scale, more useful (p=0.009). For all providers, the perceived qualities of useful at point of care, helps with decision making, saves time diagnosing, fits into thought process, and should be the standard of clinical care correlated highly (≥0.65) with overall 10-point usefulness scores.

CONCLUSIONS

Healthcare providers describe clear preferences for certain clinical prediction rules, based on medical specialty.

A Framework for Usable and Effective Clinical Decision Support: Experience from the iCPR Randomized Clinical Trial

INTRODUCTION

The promise of Clinical Decision Support (CDS) has always been to transform patient care and improve patient outcomes through the delivery of timely and appropriate recommendations that are patient specific and, more often than not, are appropriately actionable. However, the users of CDS-providers-are frequently bombarded with inappropriate and inapplicable CDS that often are not informational, not integrated into the workflow, not patient specific, and that may present out of date and irrelevant recommendations.

METHODS

The integrated clinical prediction rule (iCPR) project was a randomized clinical trial (RCT) conducted to determine if a novel form of CDS, i.e., clinical prediction rules (CPRs), could be efficiently integrated into workflow and result in changes in outcomes (e.g., antibiotic ordering) when embedded within a commercial electronic health record (EHR). We use the lessons learned from the iCPR project to illustrate a framework for constructing usable, useful, and effective actionable CDS while employing off-the-shelf functionality in a production system. Innovations that make up the framework combine the following: (1) active and actionable decision support, (2) multiple rounds of usability testing with iterative development for user acceptance, (3) numerous context sensitive triggers, (4) dedicated training and support for users of the CDS tool for user adoption, and (5) support from clinical and administrative leadership. We define "context sensitive triggers" as being workflow events (i.e., context) that result in a CDS intervention.

DISCUSSION

Success of the framework can be measured by CDS adoption (i.e., intervention is being used), acceptance (compliance with recommendations), and clinical outcomes (where appropriate). This framework may have broader implications for the deployment of Health Information Technology (HIT).

RESULTS AND CONCLUSION

iCPR was well adopted(57.4% of users) and accepted (42.7% of users). Usability testing identified and fixed many issues before the iCPR RCT. The level of leadership support and clinical guidance for iCPR was key in establishing a culture of acceptance for both the tool and its recommendations contributing to adoption and acceptance. The dedicated training and support lead to the majority of the residents reporting a high level of comfort with both iCPR tools strep pharyngitis (64.4 percent) and pneumonia (62.7 percent) as well as a high likelihood of using the tools in the future. A surprising framework addition resulted from usability testing: context sensitive triggers.

CDS, UX, and System Redesign - Promising Techniques and Tools to Bridge the Evidence Gap

Introduction:

In this special issue of eGEMs, we explore the struggles related to bringing evidence into day-to-day practice, what I define as the “evidence gap.” We are all aware of high quality evidence in the form of guidelines, randomized clinical trials for treatments and diagnostic tests, and clinical prediction rules, which are all readily available online. We also know that electronic health records (EHRs) are now ubiquitous in health care and in most practices across the country. How we marry this high quality evidence and the practice of medicine through effective decision support is a major challenge.

About the Issue:

All of the articles in this issue explore, in some fashion, CDS systems and how we can best bring providers and their work environment to the evidence. We are at the very early stages of the science of usability. Much more research and funding is needed in this area if we hope to improve the dissemination and implementation of evidence in practice. While the featured examples, techniques, and tools in the special issue are a promising start to improving usability and CDS, many of the papers highlight current gaps in knowledge and a great need for generalizable approaches. The great promise is for “learning” approaches to generate new evidence and to integrate this evidence in reliable, patient-centered ways at scale using new technology. Closing the evidence gap is a real possibility, but only if the community works together to innovate and invest in research on the best ways to disseminate, communicate, and implement evidence in practice.

External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system

Background

Hospitalized medical patients are at risk for venous thromboembolism (VTE). Universal application of pharmacological thromboprophylaxis has the potential to place a large number of patients at increased bleeding risk. In this study, we aimed to externally validate the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) VTE risk assessment model in a hospitalized general medical population.

Methods and Results

We identified medical discharges that met the IMPROVE protocol. Cases were defined as hospital‐acquired VTE and confirmed by diagnostic study within 90 days of index hospitalization; matched controls were also identified. Risk factors for VTE were based on the IMPROVE risk assessment model (aged >60 years, prior VTE, intensive care unit or coronary care unit stay, lower limb paralysis, immobility, known thrombophilia, and cancer) and were measured and assessed. A total of 19 217 patients met the inclusion criteria. The overall VTE event rate was 0.7%. The IMPROVE risk assessment model identified 2 groups of the cohort by VTE incidence rate: The low‐risk group had a VTE event rate of 0.42 (95% CI 0.31 to 0.53), corresponding to a score of 0 to 2, and the at‐risk group had a VTE event rate of 1.29 (95% CI 1.01 to 1.57), corresponding to a score of ≥3. Low‐risk status for VTE encompassed 68% of the patient cohort. The area under the receiver operating characteristic curve was 0.702, which was in line with the derivation cohort findings.

Conclusions

The IMPROVE VTE risk assessment model validation cohort revealed good discrimination and calibration for both the overall VTE risk model and the identification of low‐risk and at‐risk medical patient groups, using a risk score of ≥3. More than two thirds of the entire cohort had a score ≤2.