Design and implementation of electronic health record integrated clinical prediction rules (iCPR): a randomized trial in diverse primary care settings

21st century (clinical) decision support in nursing and allied healthcare. Developing a learning health system: a reasoned design of a theoretical framework

In this paper, we present a framework for developing a Learning Health System (LHS) to provide means to a computerized clinical decision support system for allied healthcare and/or nursing professionals. LHSs are well suited to transform healthcare systems in a mission-oriented approach, and is being adopted by an increasing number of countries. Our theoretical framework provides a blueprint for organizing such a transformation with help of evidence based state of the art methodologies and techniques to eventually optimize personalized health and healthcare. Learning via health information technologies using LHS enables users to learn both individually and collectively, and independent of their location. These developments demand healthcare innovations beyond a disease focused orientation since clinical decision making in allied healthcare and nursing is mainly based on aspects of individuals' functioning, wellbeing and (dis)abilities. Developing LHSs depends heavily on intertwined social and technological innovation, and research and development. Crucial factors may be the transformation of the Internet of Things into the Internet of FAIR data & services. However, Electronic Health Record (EHR) data is in up to 80% unstructured including free text narratives and stored in various inaccessible data warehouses. Enabling the use of data as a driver for learning is challenged by interoperability and reusability.To address technical needs, key enabling technologies are suitable to convert relevant health data into machine actionable data and to develop algorithms for computerized decision support. To enable data conversions, existing classification and terminology systems serve as definition providers for natural language processing through (un)supervised learning.To facilitate clinical reasoning and personalized healthcare using LHSs, the development of personomics and functionomics are useful in allied healthcare and nursing. Developing these omics will be determined via text and data mining. This will focus on the relationships between social, psychological, cultural, behavioral and economic determinants, and human functioning.Furthermore, multiparty collaboration is crucial to develop LHSs, and man-machine interaction studies are required to develop a functional design and prototype. During development, validation and maintenance of the LHS continuous attention for challenges like data-drift, ethical, technical and practical implementation difficulties is required.

Integrating Clinical Decision Support Into Electronic Health Record Systems Using a Novel Platform (EvidencePoint): Developmental Study

BACKGROUND

Through our work, we have demonstrated how clinical decision support (CDS) tools integrated into the electronic health record (EHR) assist providers in adopting evidence-based practices. This requires confronting technical challenges that result from relying on the EHR as the foundation for tool development; for example, the individual CDS tools need to be built independently for each different EHR.

OBJECTIVE

The objective of our research was to build and implement an EHR-agnostic platform for integrating CDS tools, which would remove the technical constraints inherent in relying on the EHR as the foundation and enable a single set of CDS tools that can work with any EHR.

METHODS

We developed EvidencePoint, a novel, cloud-based, EHR-agnostic CDS platform, and we will describe the development of EvidencePoint and the deployment of its initial CDS tools, which include EHR-integrated applications for clinical use cases such as prediction of hospitalization survival for patients with COVID-19, venous thromboembolism prophylaxis, and pulmonary embolism diagnosis.

RESULTS

The results below highlight the adoption of the CDS tools, the International Medical Prevention Registry on Venous Thromboembolism-D-Dimer, the Wells' criteria, and the Northwell COVID-19 Survival (NOCOS), following development, usability testing, and implementation. The International Medical Prevention Registry on Venous Thromboembolism-D-Dimer CDS was used in 5249 patients at the 2 clinical intervention sites. The intervention group tool adoption was 77.8% (4083/5249 possible uses). For the NOCOS tool, which was designed to assist with triaging patients with COVID-19 for hospital admission in the event of constrained hospital resources, the worst-case resourcing scenario never materialized and triaging was never required. As a result, the NOCOS tool was not frequently used, though the EvidencePoint platform's flexibility and customizability enabled the tool to be developed and deployed rapidly under the emergency conditions of the pandemic. Adoption rates for the Wells' criteria tool will be reported in a future publication.

CONCLUSIONS

The EvidencePoint system successfully demonstrated that a flexible, user-friendly platform for hosting CDS tools outside of a specific EHR is feasible. The forthcoming results of our outcomes analyses will demonstrate the adoption rate of EvidencePoint tools as well as the impact of behavioral economics "nudges" on the adoption rate. Due to the EHR-agnostic nature of EvidencePoint, the development process for additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system of EvidencePoint.

Derivation of a Risk Assessment Tool for Prediction of Long-Term Pain Intensity Reduction After Physical Therapy

Rationale

Risk assessment tools can improve clinical decision-making for individuals with musculoskeletal pain, but do not currently exist for predicting reduction of pain intensity as an outcome from physical therapy.

Aims and Objective

The objective of this study was to develop a tool that predicts failure to achieve a 50% pain intensity reduction by 1) determining the appropriate statistical model to inform the tool and 2) select the model that considers the tradeoff between clinical feasibility and statistical accuracy.

Methods

This was a retrospective, secondary data analysis of the Optimal Screening for Prediction of Referral and Outcome (OSPRO) cohort. Two hundred and seventy-nine individuals seeking physical therapy for neck, shoulder, back, or knee pain who completed 12-month follow-up were included. Two modeling approaches were taken: a longitudinal model included demographics, presence of previous episodes of pain, and regions of pain in addition to baseline and change in OSPRO Yellow Flag scores to 12 months; two comparison models included the same predictors but assessed only baseline and early change (4 weeks) scores. The primary outcome was failure to achieve a 50% reduction in pain intensity score at 12 months. We compared the area under the curve (AUC) to assess the performance of each candidate model and to determine which to inform the Personalized Pain Prediction (P3) risk assessment tool.

Results

The baseline only and early change models demonstrated lower accuracy (AUC=0.68 and 0.71, respectively) than the longitudinal model (0.79) but were within an acceptable predictive range. Therefore, both baseline and early change models were used to inform the P3 risk assessment tool.

Conclusion

The P3 tool provides physical therapists with a data-driven approach to identify patients who may be at risk for not achieving improvements in pain intensity following physical therapy.

Real-world implementation of precision psychiatry: Transdiagnostic risk calculator for the automatic detection of individuals at-risk of psychosis

BACKGROUND

Risk estimation models integrated into Electronic Health Records (EHRs) can deliver innovative approaches in psychiatry, but clinicians' endorsement and their real-world usability are unknown. This study aimed to investigate the real-world feasibility of implementing an individualised, transdiagnostic risk calculator to automatically screen EHRs and detect individuals at-risk for psychosis.

METHODS

Feasibility implementation study encompassing an in-vitro phase (March 2018 to May 2018) and in-vivo phase (May 2018 to April 2019). The in-vitro phase addressed implementation barriers and embedded the risk calculator (predictors: age, gender, ethnicity, index cluster diagnosis, age*gender) into the local EHR. The in-vivo phase investigated the real-world feasibility of screening individuals accessing secondary mental healthcare at the South London and Maudsley NHS Trust. The primary outcome was adherence of clinicians to automatic EHR screening, defined by the proportion of clinicians who responded to alerts from the risk calculator, over those contacted.

RESULTS

In-vitro phase: implementation barriers were identified/overcome with clinician and service user engagement, and the calculator was successfully integrated into the local EHR through the CogStack platform. In-vivo phase: 3722 individuals were automatically screened and 115 were detected. Clinician adherence was 74% without outreach and 85% with outreach. One-third of clinicians responded to the first email (37.1%) or phone calls (33.7%). Among those detected, cumulative risk of developing psychosis was 12% at six-month follow-up.

CONCLUSION

This is the first implementation study suggesting that combining precision psychiatry and EHR methods to improve detection of individuals with emerging psychosis is feasible. Future psychiatric implementation research is urgently needed.

Impact of Clinical Decision Support on Antibiotic Prescribing for Acute Respiratory Infections: a Cluster Randomized Implementation Trial

BACKGROUND

Clinical decision support (CDS) is a promising tool for reducing antibiotic prescribing for acute respiratory infections (ARIs).

OBJECTIVE

To assess the impact of previously effective CDS on antibiotic-prescribing rates for ARIs when adapted and implemented in diverse primary care settings.

DESIGN

Cluster randomized clinical trial (RCT) implementing a CDS tool designed to guide evidence-based evaluation and treatment of streptococcal pharyngitis and pneumonia.

SETTING

Two large academic health system primary care networks with a mix of providers.

PARTICIPANTS

All primary care practices within each health system were invited. All providers within participating clinic were considered a participant. Practices were randomized selection to a control or intervention group.

INTERVENTIONS

Intervention practice providers had access to an integrated clinical prediction rule (iCPR) system designed to determine the risk of bacterial infection from reason for visit of sore throat, cough, or upper respiratory infection and guide evidence-based evaluation and treatment.

MAIN OUTCOME(S)

Change in overall antibiotic prescription rates.

MEASURE(S)

Frequency, rates, and type of antibiotics prescribed in intervention and controls groups.

RESULTS

33 primary care practices participated with 541 providers and 100,573 patient visits. Intervention providers completed the tool in 6.9% of eligible visits. Antibiotics were prescribed in 35% and 36% of intervention and control visits, respectively, showing no statistically significant difference. There were also no differences in rates of orders for rapid streptococcal tests (RR, 0.94; P = 0.11) or chest X-rays (RR, 1.01; P = 0.999) between groups.

CONCLUSIONS

The iCPR tool was not effective in reducing antibiotic prescription rates for upper respiratory infections in diverse primary care settings. This has implications for the generalizability of CDS tools as they are adapted to heterogeneous clinical contexts.

TRIAL REGISTRATION

Clinicaltrials.gov (NCT02534987). Registered August 26, 2015 at https://clinicaltrials.gov.

Engaging stakeholders in the adaptation of the Connect for Health pediatric weight management program for national implementation

Background

Connect for Health is an evidence-based weight management program with clinical- and family-facing components for delivery in pediatric primary care for families of children ages 2 to 12 years. We used the Consolidated Framework for Implementation Research (CFIR) to guide formative work prior to national implementation. The purpose of this study was to describe the process and results of stakeholder engagement and program adaptation.

Methods

We used mixed qualitative and quantitative methods to iteratively adapt and optimize the program by assessing needs and perspectives of clinicians and parents, as well as contextual barriers, facilitators, and organizational readiness for the uptake of the proposed program tools and implementation strategies. We conducted interviews with primary care clinicians from four health care organizations in Boston, MA; Denver, CO; and Greenville, SC, and used principles of immersion-crystallization for qualitative analyses. We also conducted surveys of parents of children with a body mass index ≥ 85th percentile.

Results

We reached thematic saturation after 52 clinician interviews. Emergent themes representing the CFIR domains of intervention characteristics, outer and inner setting, and process included (1) importance of evidence-based clinical decision support tools that integrate into the workflow and do not extend visit time, (2) developing resources that respond to family’s needs, (3) using multimodal delivery options for family resources, (4) addressing childhood obesity while balancing competing demands, (5) emphasizing patient care rather than documentation and establishing sustainability plans, and (6) offering multiple training methods that incorporate performance feedback. Of the parents surveyed (n = 400), approximately 50% were Spanish-speaking and over 75% reported an annual income < $50,000. Parents affirmed the importance of addressing weight management during well-child visits, being provided with referrals and resources, and offering multiple methods for resource delivery. Decisions about program modifications were made at the program and healthcare-system level and based on stakeholder engagement findings. Modifications included cultural, geographic, and target audience adaptations, as well as varied resource delivery options.

Conclusions

To ensure the fit between the Connect for Health program and national implementation settings, adaptations were systematically made through engagement of clinician and parent stakeholders to support adoption, sustainability, and health outcomes.

Trial registration

NCT04042493

Developing a framework for evidence-based grading and assessment of predictive tools for clinical decision support

BACKGROUND

Clinical predictive tools quantify contributions of relevant patient characteristics to derive likelihood of diseases or predict clinical outcomes. When selecting predictive tools for implementation at clinical practice or for recommendation in clinical guidelines, clinicians are challenged with an overwhelming and ever-growing number of tools, most of which have never been implemented or assessed for comparative effectiveness. To overcome this challenge, we have developed a conceptual framework to Grade and Assess Predictive tools (GRASP) that can provide clinicians with a standardised, evidence-based system to support their search for and selection of efficient tools.

METHODS

A focused review of the literature was conducted to extract criteria along which tools should be evaluated. An initial framework was designed and applied to assess and grade five tools: LACE Index, Centor Score, Well's Criteria, Modified Early Warning Score, and Ottawa knee rule. After peer review, by six expert clinicians and healthcare researchers, the framework and the grading of the tools were updated.

RESULTS

GRASP framework grades predictive tools based on published evidence across three dimensions: 1) Phase of evaluation; 2) Level of evidence; and 3) Direction of evidence. The final grade of a tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports a positive conclusion. Ottawa knee rule had the highest grade since it has demonstrated positive post-implementation impact on healthcare. LACE Index had the lowest grade, having demonstrated only pre-implementation positive predictive performance.

CONCLUSION

GRASP framework builds on widely accepted concepts to provide standardised assessment and evidence-based grading of predictive tools. Unlike other methods, GRASP is based on the critical appraisal of published evidence reporting the tools' predictive performance before implementation, potential effect and usability during implementation, and their post-implementation impact. Implementing the GRASP framework as an online platform can enable clinicians and guideline developers to access standardised and structured reported evidence of existing predictive tools. However, keeping GRASP reports up-to-date would require updating tools' assessments and grades when new evidence becomes available, which can only be done efficiently by employing semi-automated methods for searching and processing the incoming information.

Using normalisation process theory to understand workflow implications of decision support implementation across diverse primary care settings

BACKGROUND

Effective implementation of technologies into clinical workflow is hampered by lack of integration into daily activities. Normalisation process theory (NPT) can be used to describe the kinds of 'work' necessary to implement and embed complex new practices. We determined the suitability of NPT to assess the facilitators, barriers and 'work' of implementation of two clinical decision support (CDS) tools across diverse care settings.

METHODS

We conducted baseline and 6-month follow-up quantitative surveys of clinic leadership at two academic institutions' primary care clinics randomised to the intervention arm of a larger study. The survey was adapted from the NPT toolkit, analysing four implementation domains: sense-making, participation, action, monitoring. Domains were summarised among completed responses (n=60) and examined by role, institution, and time.

RESULTS

The median score for each NPT domain was the same across roles and institutions at baseline, and decreased at 6 months. At 6 months, clinic managers' participation domain (p=0.003), and all domains for medical directors (p<0.003) declined. At 6 months, the action domain decreased among Utah respondents (p=0.03), and all domains decreased among Wisconsin respondents (p≤0.008).

CONCLUSIONS

This study employed NPT to longitudinally assess the implementation barriers of new CDS. The consistency of results across participant roles suggests similarities in the work each role took on during implementation. The decline in engagement over time suggests the need for more frequent contact to maintain momentum. Using NPT to evaluate this implementation provides insight into domains which can be addressed with participants to improve success of new electronic health record technologies.

TRIAL REGISTRATION NUMBER

NCT02534987.

User-Centered Development of a Behavioral Economics Inspired Electronic Health Record Clinical Decision Support Module

Changing physician behaviors is difficult. Electronic health record (EHR) clinical decision support (CDS) offers an opportunity to promote guideline adherence. Behavioral economics (BE) has shown success as an approach to supporting evidence-based decision-making with little additional cognitive burden. We applied a user-centered approach to incorporate BE “nudges” into a CDS module in two “vanguard” sites utilizing: (1) semi-structured interviews with key informants (n=8); (2) a design thinking workshop; and (3) semi-structured group interviews with clinicians. In the 133 day development phase at two clinics, the navigator section fired 299 times for 27 unique clinicians. The inbasket refill alert fired 124 times for 22 clinicians. Fifteen prescriptions for metformin were written by 11 clinicians. Our user-centered approach yielded a BE- driven CDS module with relatively high utilization by clinicians. Next steps include the addition of two modules and continued tracking of utilization, and assessment of clinical impact of the module.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Live Usability Testing of Two Complex Clinical Decision Support Tools: Observational Study

BACKGROUND

Potential of the electronic health records (EHR) and clinical decision support (CDS) systems to improve the practice of medicine has been tempered by poor design and the resulting burden they place on providers. CDS is rarely tested in the real clinical environment. As a result, many tools are hard to use, placing strain on providers and resulting in low adoption rates. The existing CDS usability literature relies primarily on expert opinion and provider feedback via survey. This is the first study to evaluate CDS usability and the provider-computer-patient interaction with complex CDS in the real clinical environment.

OBJECTIVE

This study aimed to further understand the barriers and facilitators of meaningful CDS usage within a real clinical context.

METHODS

This qualitative observational study was conducted with 3 primary care providers during 6 patient care sessions. In patients with the chief complaint of sore throat, a CDS tool built with the Centor Score was used to stratify the risk of group A Streptococcus pharyngitis. In patients with a chief complaint of cough or upper respiratory tract infection, a CDS tool built with the Heckerling Rule was used to stratify the risk of pneumonia. During usability testing, all human-computer interactions, including audio and continuous screen capture, were recorded using the Camtasia software. Participants' comments and interactions with the tool during clinical sessions and participant comments during a postsession brief interview were placed into coding categories and analyzed for generalizable themes.

RESULTS

In the 6 encounters observed, primary care providers toggled between addressing either the computer or the patient during the visit. Minimal time was spent listening to the patient without engaging the EHR. Participants mostly used the CDS tool with the patient, asking questions to populate the calculator and discussing the results of the risk assessment; they reported the ability to do this as the major benefit of the tool. All providers were interrupted during their use of the CDS tool by the need to refer to other sections of the chart. In half of the visits, patients' clinical symptoms challenged the applicability of the tool to calculate the risk of bacterial infection. Primary care providers rarely used the incorporated incentives for CDS usage, including progress notes and patient instructions.

CONCLUSIONS

Live usability testing of these CDS tools generated insights about their role in the patient-provider interaction. CDS may contribute to the interaction by being simultaneously viewed by the provider and patient. CDS can improve usability and lessen the strain it places on providers by being short, flexible, and customizable to unique provider workflow. A useful component of CDS is being as widely applicable as possible and ensuring that its functions represent the fastest way to perform a particular task.

Validation of a preoperative prediction model for mortality within 1 year after endovascular aortic aneurysm repair of intact aneurysms

OBJECTIVE

Most would agree that at least 1-year survival is necessary after intact abdominal aortic aneurysm (AAA) repair to appropriately justify the cost and risk of the procedure. No validated clinical decision instruments exist to predict survival after endovascular aneurysm repair (EVAR) beyond the perioperative period. The purpose of this analysis was to create a preoperative prediction model for 1-year mortality after EVAR for intact AAA in the Society for Vascular Surgery Vascular Quality Initiative.

METHODS

All intact EVARs in the Society for Vascular Surgery Vascular Quality Initiative from 2011 to 2015 were randomly divided into training (n = 17,836) and validation (n = 2500) data sets, and 31 preoperative candidate predictors were identified. A logistic regression model for 1-year mortality was created, and bootstrapped stepwise variable elimination was used to reduce this model to a best subset of predictors. Penalized maximum likelihood estimation was used to correct for potential overfitting. The final model was internally validated by bootstrapping the area under the curve (AUC) and the calibration slope and intercept, and its performance when applied to the training and validation data sets was compared.

RESULTS

After elective and nonelective (symptomatic, intact) EVAR, 1-year mortality was 5.5% (n = 900/16,411) and 11.4% (n = 162/1425), respectively. The mean probability of 1-year mortality was 6.0% (n = 1062) in the training set and 5.7% (n = 143) in the validation cohort (P = .12). Significant preoperative predictors of 1-year mortality included chronic obstructive pulmonary disease, age, preoperative renal insufficiency (creatinine concentration ≥1.8 mg/dL or on hemodialysis), ejection fraction <50%, transfer status, body mass index <24 kg/m², preoperative beta-blocker exposure, larger AAA diameter, and lower admission hemoglobin level. Preoperative statin use was found to be protective. The bias-corrected AUC was 0.759 (Hosmer-Lemeshow goodness-of-fit P value of 0.36; calibration intercept, -0.003; slope, 0.999). When applied to the validation data set, the model had AUC of 0.724 (95% confidence interval, 0.676-0.768; calibration intercept, 0.0009; slope, 0.970), which was in excellent agreement with the original data set bias-corrected AUC. Notably, ∼27.5% (n = 4902) had four or more risk factors with a predicted 1-year post-EVAR mortality risk of 10% to 22% despite that 33.2% of these patients had AAA diameters below recommended treatment guideline minimum thresholds.

CONCLUSIONS

This validated preoperative prediction model for 1-year mortality identifies patients less likely to benefit from EVAR. Appropriateness of intact AAA EVAR care delivery can be improved by use of this clinical decision aid to determine which high-risk patients have lower probability of mortality within the first postoperative year relative to their predicted annualized rupture risk.

Designing for implementation: user-centered development and pilot testing of a behavioral economic-inspired electronic health record clinical decision support module

Background

Current guidelines recommend less aggressive target hemoglobin A1c (HbA1c) levels based on older age and lower life expectancy for older adults with diabetes. The effectiveness of electronic health record (EHR) clinical decision support (CDS) in promoting guideline adherence is undermined by alert fatigue and poor workflow integration. Integrating behavioral economics (BE) and CDS tools is a novel approach to improving adherence to guidelines while minimizing clinician burden.

Methods

We will apply a systematic, user-centered design approach to incorporate BE “nudges” into a CDS module and will perform user testing in two “vanguard” sites. To accomplish this, we will conduct (1) semi-structured interviews with key informants (n = 8), (2) a 2-h, design-thinking workshop to derive and refine initial module ideas, and (3) semi-structured group interviews at each site with clinic leaders and clinicians to elicit feedback on three proposed nudge module components (navigator section, inbasket refill protocol, medication preference list). Detailed field notes will be summarized by module idea and usability theme for rapid iteration. Frequency of firing and user action taken will be assessed in the first month of implementation via EHR reporting to confirm that module components and related reporting are working as expected as well as assess utilization. To assess the utilization and feasibility of the new tools and generate estimates of clinician compliance with the Choosing Wisely guideline for diabetes management in older adults, a 6-month, single-arm pilot study of the BE-EHR module will be conducted in six outpatient primary care clinics.

Discussion

We hypothesize that a low burden, user-centered approach to design will yield a BE-driven, CDS module with relatively high utilization by clinicians. The resulting module will establish a platform for exploring the ability of BE concepts embedded within the EHR to affect guideline adherence for other use cases.

Electronic supplementary material

The online version of this article (10.1186/s40814-019-0403-z) contains supplementary material, which is available to authorized users.

Adaptive design of a clinical decision support tool: What the impact on utilization rates means for future CDS research

OBJECTIVE

We employed an agile, user-centered approach to the design of a clinical decision support tool in our prior integrated clinical prediction rule study, which achieved high adoption rates. To understand if applying this user-centered process to adapt clinical decision support tools is effective in improving the use of clinical prediction rules, we examined utilization rates of a clinical decision support tool adapted from the original integrated clinical prediction rule study tool to determine if applying this user-centered process to design yields enhanced utilization rates similar to the integrated clinical prediction rule study.

MATERIALS & METHODS: We conducted pre-deployment usability testing and semi-structured group interviews at 6 months post-deployment with 75 providers at 14 intervention clinics across the two sites to collect user feedback. Qualitative data analysis is bifurcated into immediate and delayed stages; we reported on immediate-stage findings from real-time field notes used to generate a set of rapid, pragmatic recommendations for iterative refinement. Monthly utilization rates were calculated and examined over 12 months.

RESULTS

We hypothesized a well-validated, user-centered clinical decision support tool would lead to relatively high adoption rates. Then 6 months post-deployment, integrated clinical prediction rule study tool utilization rates were substantially lower than anticipated based on the original integrated clinical prediction rule study trial (68%) at 17% (Health System A) and 5% (Health System B). User feedback at 6 months resulted in recommendations for tool refinement, which were incorporated when possible into tool design; however, utilization rates at 12 months post-deployment remained low at 14% and 4% respectively.

DISCUSSION

Although valuable, findings demonstrate the limitations of a user-centered approach given the complexity of clinical decision support.

CONCLUSION

Strategies for addressing persistent external factors impacting clinical decision support adoption should be considered in addition to the user-centered design and implementation of clinical decision support.

THE HIGH COST OF LOW VALUE CARE

The main focus of this study is bridging the "evidence gap" between frontline decision-making in health care and the actual evidence, with the hope of reducing unnecessary diagnostic testing and treatments. From our work in pulmonary embolism (PE) and over ordering of computed tomography pulmonary angiography, we integrated the highly validated Wells' criteria into the electronic health record at two of our major academic tertiary hospitals. The Wells' clinical decision support tool triggered for patients being evaluated for PE and therefore determined a patients' pretest probability for having a PE. There were 12,759 patient visits representing 11,836 patients, 51% had no D-dimer, 41% had a negative D-dimer, and 9% had a positive D-dimer. Our study gave us an opportunity to determine which patients were very low probabilities for PE, with no need for further testing.

Innovating From Within: A Process Model for User-Centered Digital Development in Academic Medical Centers

BACKGROUND

Design thinking and human-centered design approaches have become increasingly common in health care literature, particularly in relation to health information technology (HIT), as a pathway toward the development of usable, diffusible tools and processes. There is a need in academic medical centers tasked with digital innovation for a comprehensive process model to guide development that incorporates current industry trends, including design thinking and lean and agile approaches to digital development.

OBJECTIVE

This study aims to describe the foundations and phases of our model for user-centered HIT development.

METHODS

Based on our experience, we established an integrated approach and rigorous process for HIT development that leverages design thinking and lean and agile strategies in a pragmatic way while preserving methodological integrity in support of academic research goals.

RESULTS

A four-phased pragmatic process model was developed for user-centered digital development in HIT.

CONCLUSIONS

The model for user-centered HIT development that we developed is the culmination of diverse innovation projects and represents a multiphased, high-fidelity process for making more creative, flexible, efficient, and effective tools. This model is a critical step in building a rigorous approach to HIT design that incorporates a multidisciplinary, pragmatic perspective combined with academic research practices and state-of-the-art approaches to digital product development to meet the unique needs of health care.

Practical and Conceptual Considerations for the Primary Prevention of Asthma

Asthma is among the most common chronic diseases worldwide and is a significant contributor to the global health burden, highlighting the urgent need for primary prevention. This article outlines several practical and conceptual challenges that accompany primary prevention efforts. It advocates for improved predictive modeling to identify those at high-risk of developing asthma using automated algorithms within electronic medical records systems and explanatory modeling to refine understanding of causal pathways. Understanding the many issues that are likely to affect the success of primary prevention efforts helps the community of individuals invested in asthma prevention organize efforts and maximize their impact.

Bridging the Gap Between Academic Research and Pragmatic Needs in Usability: A Hybrid Approach to Usability Evaluation of Health Care Information Systems

BACKGROUND

Technology is increasingly embedded into the full spectrum of health care. This movement has benefited from the application of software development practices such as usability testing and agile development processes. These practices are frequently applied in both commercial or operational and academic settings. However, the relative importance placed on rapid iteration, validity, reproducibility, generalizability, and efficiency differs between the 2 settings and the needs and objectives of academic versus pragmatic usability evaluations.

OBJECTIVE

This paper explores how usability evaluation typically varies on key dimensions in pragmatic versus academic settings that impact the rapidity, validity, and reproducibility of findings and proposes a hybrid approach aimed at satisfying both pragmatic and academic objectives.

METHODS

We outline the characteristics of pragmatic versus academically oriented usability testing in health care, describe the tensions and gaps resulting from differing contexts and goals, and present a model of this hybrid process along with 2 case studies of digital development projects in which we demonstrate this integrated approach to usability evaluation.

RESULTS

The case studies presented illustrate design choices characteristic of our hybrid approach to usability evaluation.

CONCLUSIONS

Designed to leverage the strengths of both pragmatically and academically focused usability studies, a hybrid approach allows new development projects to efficiently iterate and optimize from usability data as well as preserves the ability of these projects to produce deeper insights via thorough qualitative analysis to inform further tool development and usability research by way of academically focused dissemination.

Using Clinical Decision Support Within the Electronic Health Record to Reduce Incorrect Prescribing for Acute Sinusitis

Purpose

Acute sinusitis has viral etiology in more than 90% of cases, but antibiotics are prescribed for more than 80% of adults in the United States. While applications of computer-assisted guidelines have been found effective in reducing inaccurate prescribing for acute respiratory infections, there is a paucity of research focused specifically on the utilization of electronic best practice alerts (BPA) in improving treatment for acute sinusitis.

Methods

This observational cohort study examined prescribing behavior for sinusitis at a single Federally Qualified Health Center 1 year prior and during the first year of implementation of a BPA in the electronic health record (EHR) reminding providers of the recommended treatment of sinusitis. The advisory included a link to national guidelines and a note template was installed to assist providers in documentation. The BPA appeared on the providers' screen when an ICD-9 code of acute or bacterial sinusitis was entered during the patient visit.

Results

After adjusting for select patient and provider factors, the computer-assisted guidelines effectively reduced the overall antibiotic prescribing among these patients by 31% (relative risk: 0.69, 95% confidence interval: 0.51-0.95) and reduced incorrect prescribing from 88.5% to 78.7% (P=0.02).

Conclusions

Clinical reminders within the EHR can be an effective tool to reduce inappropriate antibiotic use and improve providers' decisions regarding the correct antibiotic choices for patients with acute sinusitis.

"Think aloud" and "Near live" usability testing of two complex clinical decision support tools

OBJECTIVES

Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. "Think Aloud" and "Near Live" usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to "Near Live" testing after completing "Think Aloud".

METHODS

This was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During "Think Aloud" testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During "Near Live" testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods.

RESULTS

"Think Aloud" and "Near Live" usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during "Near Live" testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during "Near-Live" testing because they would not be all inclusive for the visit.

CONCLUSIONS

These complementary types of usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. The additional feedback from "Near Live" testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.